Analyzing Job Posting Data for Business Insights

Every job posting is a small, public, timestamped statement of intent: a specific company wants a specific kind of person in a specific place, now. Aggregated over thousands of companies and months of data, those statements describe labor demand with a resolution that official statistics rarely reach - by job title, skill, city and employer, updated daily.

This guide is the methodical version of "use job data for business insights". It covers which questions job postings can and cannot answer, what a posting record actually contains, a repeatable analysis workflow, the metrics that work, the biases that trip people up, and the tooling to do it on your own machine.

Which Business Questions Job Posting Data Answers

Job postings measure demand for labor as expressed by employers. That makes them a good fit for questions about where, what and who is hiring - and a poor fit for questions about actual hires or headcount (more on that under pitfalls).

A few patterns behind these use cases:

• Sales intelligence. A company that posts for "Salesforce administrator" probably uses Salesforce; one that posts ten warehouse roles in a new city is probably opening a site. Both are buying signals for vendors.
• Competitive intelligence. Postings often name the team, the tech stack and the product line. A competitor suddenly hiring ML engineers in a market where it had none is a strategic signal months before a press release.
• Hiring strategy. Comparing your own postings with those of companies competing for the same candidates shows how titles, requirements and offered benefits differ.

What a Job Posting Record Contains

Before designing any analysis, know your fields and how often they are filled. Techmap delivers postings as gzip-compressed JSON Lines files, one file per country per day, with one JSON object per posting. The full export schema is documented on the Job Data Overview; the most relevant fields for analysis are:

Two consequences for analysis:

1. Salary analysis runs on a subset. With roughly a quarter of postings carrying salary data, any salary statistic describes "postings that state a salary", which skews toward countries and roles where pay transparency is common or required.
2. Fields are "as stated". Company names, locations and titles come from the original posting. Normalization is your job (or your pipeline's), which is why the cleaning step below matters.

The Jobs API exposes additional derived fields and filters - for example industry, workPlace (remote, hybrid, onsite - only when stated in the posting), careerLevel and an isDuplicate flag.

An Analysis Workflow in Five Steps

The steps below are the same whether you analyze one country for one month or ten countries since 2020. For a more detailed walk-through, see the step-by-step guide to job posting data analysis.

1. Define the question precisely

"What is happening in the tech job market?" is not analyzable. "How did the monthly number of new postings for data engineers in Germany, Austria and Switzerland change between 2023 and 2026, and which employers drove it?" is. A good question fixes:

• the unit (new postings, active postings, distinct companies),
• the population (roles, skills, industries, companies),
• the geography and the time range,
• the comparison (year over year, against a baseline, against competitors).

2. Select countries, sources and time range

Pick countries by your question, not by availability, and check coverage before trusting a trend. Techmap's countries list and source list show what is collected where; currently that is 250 countries and territories and more than 200 sources, including ATS career pages, job boards, aggregators and public employment offices.

Choose the time range to cover at least one full seasonal cycle for any trend claim - twelve months minimum, more if you want to separate a trend from a one-off year. Historical data from January 2020 onward lets you compare against pre-2022 baselines.

3. Clean and normalize

This is where most of the work is, and where most bad conclusions originate.

• Deduplicate. The same job appears on the employer's career page, on several job boards and on aggregators. Techmap's API flags a posting as a duplicate when company name, city, country and title match within a 3-day window, and points to the earliest posting. In raw files you can apply the same logic yourself. Keep duplicates in a separate table rather than deleting them: how widely a company distributes its ads is itself a signal of hiring pressure and recruiting budget.
• Normalize job titles. "Sr. Software Engineer (m/w/d)", "Senior Software Developer - Backend" and "Software Engineer III" should end up in the same bucket. Strip gender markers and noise, lowercase, and map titles to an occupation taxonomy (e.g. ISCO, O*NET/SOC or your own) via rules, embeddings or a classifier.
• Normalize companies. Merge legal-form variants ("Acme GmbH", "ACME", "Acme Group") and separate staffing agencies from direct employers if your question is about end employers.
• Normalize locations. Map free-text locations to a consistent city/region/country hierarchy; treat "Remote" as its own value, not as a missing city.
• Handle reposts and expiry. Employers re-publish the same job to push it back to the top of job boards. Detect reposts by matching on company, normalized title and location over a longer window (e.g. 30-60 days) and decide whether your metric counts the job once or each publication. For expiry, use a stated end date (schema.org validThrough) where present, otherwise an assumed lifetime such as 30 days after dateCreated.

4. Aggregate

Aggregate to the unit defined in step 1: counts per week or month, per occupation, region and company. Calculate shares and indices rather than only absolute counts - shares are more robust against changes in source coverage.

5. Interpret

Compare against a baseline (same month last year, the overall market, a peer group), check whether a movement is driven by a few large employers, and sanity-check with a sample of the actual postings. Then write down the caveats: coverage, missing fields, and what the data does not show.

For more advanced methods such as clustering, NLP on posting texts and classification, see job data mining techniques and best practices.

Metrics That Work

Absolute volumes such as "postings per month" are useful for orientation (Techmap collects about 8 million new job postings per month and about 12 million active postings per month across all countries), but for comparisons over time, ratios and indexed series (e.g. January 2024 = 100) are safer.

Pitfalls and Biases

Source coverage. A dataset only contains what its sources publish. Some industries hire mainly through referrals, agencies or print, some countries rely on one dominant job board, and sources get added over time. A jump in postings can simply mean a new source. Check trends per source and use shares where possible.

Duplicates and reposts. Without deduplication, a company that spreads one job across ten portals looks ten times as active as one that posts only on its career page. Without repost handling, an evergreen job counts twelve times a year.

Ghost and evergreen jobs. Some postings collect applications for a talent pool ("always hiring drivers"), stay online after the position is filled, or are posted for compliance reasons. Long lifetimes and frequent reposts of identical texts are typical markers. Flag them rather than silently dropping them.

Seasonality. Posting volumes follow the calendar: holiday periods are quiet, the start of the year and the end of summer are busy in many countries, and retail or agriculture have their own cycles. Compare year over year or seasonally adjust before claiming a trend.

A posting is not a hire. Postings measure intent, not outcomes. One posting can stand for several openings ("we hire 20 warehouse workers"), and many openings are filled without any posting. Use postings as a leading indicator of demand, not as a headcount figure.

Missing and "as stated" fields. Salary, remote status and career level are only present when the employer states them. A rising remote share can reflect more employers labeling remote work, not more remote work.

Tools: SQL, pandas, DuckDB and BI Dashboards

You do not need a cluster for most job data analyses. One country-month of postings fits comfortably on a laptop; multi-year, multi-country work benefits from a columnar format and a warehouse.

• DuckDB queries gzip-compressed JSON Lines files directly with SQL - ideal for exploration and medium-size aggregations.
• pandas or Polars for custom cleaning, text processing and feature engineering in Python.
• Parquet as the storage format once you query the same data repeatedly - see how to convert job postings in JSON to Parquet.
• Warehouses (BigQuery, Snowflake, Redshift, Athena) for multi-year, multi-country history and team access.
• BI tools (Looker Studio, Power BI, Tableau, Metabase, Superset) for dashboards on the aggregated tables. See building a job data insights dashboard and job posting data visualization.

Example: company hiring velocity with DuckDB

The following query reads a month of daily feed files for one country, removes duplicates with the same company, normalized title and city within a week, and ranks companies by new postings. It is an illustrative sketch; adjust field paths to the schema documented on the Job Data Overview.

-- DuckDB: read gzip-compressed JSON Lines directly
CREATE TABLE jobs AS
SELECT
  idInSource,
  source,
  sourceCC                                   AS country,
  CAST(dateCreated AS TIMESTAMP)             AS created,
  lower(trim(name))                          AS title_norm,
  lower(trim(company.name))                  AS company_norm,
  lower(trim(location.orgAddress.city))      AS city_norm
FROM read_json_auto('techmap-jobs_de_2026-09-*.json.gz',
                    format = 'newline_delimited',
                    union_by_name = true);

-- Keep the first posting per company / title / city / week
WITH dedup AS (
  SELECT *
  FROM jobs
  QUALIFY row_number() OVER (
    PARTITION BY company_norm, title_norm, city_norm, date_trunc('week', created)
    ORDER BY created
  ) = 1
)
SELECT company_norm,
       count(*)                   AS new_postings,
       count(DISTINCT city_norm)  AS cities,
       count(DISTINCT source)     AS sources_used
FROM dedup
WHERE company_norm IS NOT NULL
GROUP BY company_norm
ORDER BY new_postings DESC
LIMIT 25;

Run the same query over consecutive months and compare to get hiring velocity; add a check for cities where a company has never posted before to get expansion signals.

Example: skill frequency with pandas

Skill frequency can be computed from orgTags where skills are tagged, or from the description text. A simple text-based version:

import pandas as pd

df = pd.read_json("techmap-jobs_us_2026-09-01.json.gz", lines=True)

skills = {
    "python": r"\bpython\b",
    "sql": r"\bsql\b",
    "aws": r"\baws\b|amazon web services",
    "kubernetes": r"\bkubernetes\b|\bk8s\b",
    "power bi": r"\bpower ?bi\b",
}

text = df["text"].fillna("").str.lower()
share = {name: text.str.contains(rx, regex=True).mean() for name, rx in skills.items()}
print(pd.Series(share).sort_values(ascending=False).round(3))

Restrict this to a normalized occupation first (e.g. only data roles) - a skill's share across all postings mostly reflects the occupation mix. For tracking these numbers over time, the post on how job ads data uncovers market trends has more examples.

How to Get the Data

Collecting postings yourself means building and maintaining crawlers for hundreds of portals, handling blocking and layout changes, and running deduplication - before the first analysis. Buying the data shifts that effort to the provider. Techmap offers three access paths, matched to analysis needs:

Historical datasets cost $2,400 - $4,800 per country, one-time; data feeds cost $400 per country and month; the API costs about $1 per 1,000 job postings with paid plans from $29/month. Since January 2020, more than 451 million new job postings have been collected. The data overview breaks down coverage by country and source.

A practical setup for most analytics teams: load the historical dataset for your countries once to build baselines and validate the cleaning pipeline, then subscribe to the feed so new files land in your S3 bucket and your dashboards update automatically.

Checklist

• Question defined with unit, population, geography, time range and comparison
• Coverage checked per country and source; at least twelve months for trend claims
• Duplicates flagged, reposts handled, expiry rule documented
• Titles, companies and locations normalized; agencies identified
• Metrics expressed as shares or indices where coverage may change
• Seasonality controlled; results checked against sample postings
• Caveats written down: posting is not a hire, sparse fields, source bias