Hope Springs Eternal - As Do Salaries? A Spring Break Project

Spring Break is often seen as a time for relaxation and fun, but it can also be an opportunity to learn something new and boost your skills. This tutorial presents a Spring Break project that combines data analysis with JavaScript programming, focusing on a topic that's always relevant for computer science students: salaries!

We'll use the Daily International Job Postings API (accessible through RapidAPI) to gather real-world salary data and then analyze it using JavaScript. This project will not only give you insights into salary trends but also enhance your programming abilities by working with APIs, processing JSON data, and performing basic statistical analysis.

Why Analyze Salaries?

Understanding salary trends is vital for setting realistic expectations of your earning potential. Informed negotiations become possible as you grasp the true market value of your skills, leading to fair compensation. Additionally, this knowledge aids you in making strategic career decisions, enabling you to choose paths with strong earning potential.

In this project, we will analyze real salary data, and you'll gain valuable, practical programming skills. We're using JavaScript to dive into the data, learn how to work with APIs, and perform basic statistical analysis.

For this tutorial, we’ll analyze salaries of Software Developer jobs in the United States from the last month:

• countryCode=us (Jobs in the USA)
• occupation=programmer (Jobs as programmer incl. all synonyms such as Software Engineer, ...)
• dateCreated=2025-02 (Jobs from last month)
• hasSalary=true (Only jobs with salaries)

Project Setup

Before we begin, ensure you have these prerequisites: basic JavaScript knowledge (variables, functions, loops, arrays), node.js and npm (install from nodejs.org), a text editor or IDE, a RapidAPI account (rapidapi.com), and an API Key for the Daily International Job Postings API (obtained after subscribing to the API).

These tools are essential for accessing and analyzing salary data. JavaScript skills will allow you to manipulate the API responses, while Node.js and npm will help manage your project dependencies. Your RapidAPI account and API key are needed to connect to the Job Posting API. With these in hand, you're ready to start analyzing salary trends.

Installing Dependencies

1. Create a new project directory:

   mkdir salary-analysis
   cd salary-analysis
   

2. Initialize a Node.js project:

   npm init -y
   

3. Install the necessary packages:

   npm install node-fetch dotenv
   

   • node-fetch: For making API requests.
   • dotenv: For managing environment variables (API keys).

Creating a .env file

Create a .env file in your project directory and add your RapidAPI key:

RAPIDAPI_KEY=your_api_key

Fetching Salary Data

Create a JavaScript file (e.g., salaryAnalysis.js) and add the following code:

require('dotenv').config();
const fetch = require('node-fetch');

const API_URL = 'https://daily-international-job-postings.p.rapidapi.com/api/v2/jobs/search';
const API_KEY = process.env.RAPIDAPI_KEY;

async function fetchJobPostings(baseParams) {
    let allJobs = [];
    let page = 1;
    let totalCount = Infinity; // Initialized to infinity to enter the loop
    const MAX_PAGES = 1; // Maximum number of pages to fetch

    while (allJobs.length < totalCount && page <= MAX_PAGES) {
        const params = `${baseParams}&page=${page}`;
        const url = `${API_URL}?${params}`;
        const options = {
            method: 'GET',
            headers: {
                'X-RapidAPI-Key': API_KEY,
                'X-RapidAPI-Host': 'daily-international-job-postings.p.rapidapi.com'
            }
        };

        try {
            const response = await fetch(url, options);
            const data = await response.json();

            totalCount = data.totalCount;
            if (!data.result) break;
            allJobs = allJobs.concat(data.result);  // Accumulate jobs
            page++;
        } catch (error) {
            console.error(`Error fetching jobs for page ${page}:`, error);
            break; // Exit loop on error
        }
    }
    console.log(`Fetched ${allJobs.length} jobs of ${totalCount} in total from ${page - 1} pages for query ${baseParams}.`);

    return allJobs; // Return all fetched jobs
}

function formatJobPostings(jobPostings) {
    let tsv = 'Occupation \t Min. Salary \t Currency \t Location \t Title \t URL\n'.replace(' \t ', '\t');

    jobPostings.forEach(job => {
        tsv += `${job.occupation}\t`;
        tsv += `${String((job.minSalary || 0).toLocaleString('en-US')).padStart(10, ' ')}\t`;
        tsv += `${job.jsonLD.salaryCurrency}\t`;
        tsv += `${String(job.city + ', ' + job.state || '').padStart(25, ' ')}\t`;
        tsv += `${String(job.title || '').padEnd(40, ' ')}\t`;
        tsv += `${job.jsonLD.url}\t`;
        tsv += `${job.jsonLD.identifier}\t`;
        tsv += `\n`;
    });

    return tsv;
}

// Example usage: Fetch salaries for software developers in the US with salary information
async function main() {
    const d = new Date();
    d.setMonth(d.getMonth() - 1);
    let lastMonth = d.toISOString().slice(0, 7);
    
    const query = [
        `dateCreated=${lastMonth}`,
        "countryCode=us",
        "occupation=programmer",
        "minSalary=20000", // Workaround to cut out hourly and monthly salaries
        "hasSalary=true"
    ]
    const queryParams = query.join("&"); // Change as needed
    const jobsWithSalaries = await fetchJobPostings(queryParams);
    const salaryTSV = await formatJobPostings(jobsWithSalaries);
    // console.log(salaryTSV)
    const filename = 'salaries.tsv'
    await fs.writeFile(filename, salaryTSV);
    console.log(`Salary data successfully written to ${filename}`);

    // Exclude salaries below 20000 or above 500000 (probably false data or refering to hourly, weekly, or project budget - but sometimes correct!)
    const outlierThreshold = 500000
    const filteredSalaries = jobsWithSalaries.map(job => (job.minSalary || 0))
                                             .filter(salary => 20000 <= salary && salary <= outlierThreshold);

    const averageSalary = Math.floor(filteredSalaries.reduce((sum, salary) => sum + salary, 0) / filteredSalaries.length);
    console.log('Average yearly min. salary :', String((averageSalary).toLocaleString('en-US')).padStart(10, ' '));

    const highestSalary = Math.max(...filteredSalaries);
    const lowestSalary  = Math.min(...filteredSalaries);
    console.log('Highest yearly min. salary :', String(highestSalary.toLocaleString('en-US')).padStart(10, ' '));
    console.log('Lowest yearly min. salary  :', String(lowestSalary .toLocaleString('en-US')).padStart(10, ' '));

    const outlierSalaries = jobsWithSalaries.map(job => (job.minSalary || 0))
                                            .filter(salary => salary > outlierThreshold);
    const outliers = outlierSalaries.length;
    console.log(`Outliers above ${outlierThreshold.toLocaleString('en-US')}     :`, String(outliers.toLocaleString('en-US')).padStart(10, ' '));
}

main();

This code defines several functions to retrieves job postings from the API, creates a TSV file, and analyze jobs with salary information for software developers in the US. The functions have the following task:

• fetchJobPostings(baseParams): This asynchronous function fetches job postings from an external API, using the provided query parameters, and handles pagination to retrieve all available results within a specified maximum number of pages, returning an array of job posting objects.
• formatJobPostings(jobPostings): This function takes an array of job posting objects and transforms them into a tab-separated value (TSV) string, formatting specific job details like occupation, salary, location, and title for easy data export.
• main(): The main function orchestrates the entire process: it constructs a query to fetch relevant job postings with salary information from the previous month, retrieves and formats the data, writes it to a TSV file, and then performs statistical analysis on the salary data, printing the results to the console.

The analyses currently performed within the main() function offer a foundational glimpse into the salary data, focusing on core statistical measures. However, the structure of the function is designed to be extensible, allowing you to incorporate a wider range of calculations or data visualizations to deepen your understanding of the salary trends.

• The averageSalary represents the calculated mean of the minimum yearly salaries found within the filtered job postings, providing an understanding of the typical salary range in the dataset.
• The highestSalary displays the maximum minimum yearly salary found among the filtered job postings, indicating the upper limit of the salary range within the analyzed data.
• The lowestSalary reveals the minimum minimum yearly salary found within the filtered job postings, establishing the lower bound of the salary range observed.
• The outlierSalaries count indicates the number of job postings with minimum yearly salaries exceeding the defined threshold, highlighting potential anomalies or exceptionally high salaries in the dataset.

NOTE: Due to the API's constraint of 25 free monthly requests, each request should be carefully crafted to maximize data retrieval within the limited quota. To avoid exceeding the limit, consider scheduling your data collection strategically, perhaps once a month, and refine your queries to target the most relevant job postings.

Summary

This tutorial guided you through creating a JavaScript tool that leveraged a Job Posting API to analyze salary trends. You learned how to fetch, filter, and statistically analyze job data, then export your findings as a TSV file.

Future enhancements could include implementing advanced statistical analyses, visualizing data, or even incorporating machine learning to predict salaries based on job characteristics. You might also expand the project by comparing salaries across different countries or occupations or analyzing the relationship between salaries and factors like specific skills.

We hope this Spring Break project was both engaging and informative, providing valuable insights into the job market and helping you make informed decisions about your future career prospects.