Job Data Mining: Techniques and Best Practices

A job posting is a semi-structured document: a title, a company, a location and a date, wrapped around a few hundred words of free text that mix duties, requirements, benefits and boilerplate. Job data mining is the work of turning millions of those documents into consistent, queryable records - and then finding patterns in them that no single posting reveals.

This guide walks through that pipeline stage by stage and notes which steps Techmap's data already covers, so you can decide what to build yourself.

The pipeline at a glance

Errors compound: a geocoder that misplaces postings distorts every regional trend computed later, so measure quality per stage, not only at the end.

1. Getting the data: scraping vs. APIs and feeds

There are three ways to obtain job postings in volume.

Scraping job boards and career pages. Crawlers discover posting URLs (sitemaps, listing pages, pagination) and fetch each detail page, typically with Scrapy, Beautiful Soup or a headless browser such as Playwright. Two aspects deserve an explicit look:

• Legal and terms of service. Many job boards restrict automated access in their terms or robots.txt, and postings can contain personal data such as recruiter names, emails and phone numbers, which brings data protection law (e.g. GDPR) into play. Review this per source; the answer differs by site and jurisdiction.
• Maintenance cost. Crawlers break when a site changes its markup, adds bot protection or moves to client-side rendering. The first crawler is cheap; keeping hundreds running and monitored is an ongoing engineering budget, plus proxy, storage and compute.

Source APIs. Some job boards and ATS platforms offer APIs or XML feeds. They are more stable than scraping, but each has its own schema, rate limits and coverage - one adapter per source.

Aggregated APIs and data feeds. A provider crawls the sources, merges them into one schema and delivers the result. Techmap, for example, collects postings from 200+ sources - ATS career pages, job boards, aggregators and public employment offices - in 250 countries and territories. The Job Posting API on RapidAPI covers the last three months; data feeds on AWS Data Exchange deliver one gzip-compressed JSON Lines file per country and day; historical datasets go back to January 2020. For a closer look at the source types and their trade-offs, see exploring job posting sources.

Whatever route you choose, store the raw payload. You will re-parse it more often than you expect.

2. Parsing: from HTML to text

A detail page contains the posting plus navigation, cookie banners, "similar jobs" widgets and footers. Parsing has three goals:

1. Extract embedded structured data first. Many pages carry a schema.org JobPosting object in JSON-LD (it is what Google for Jobs reads). When present, it gives you title, datePosted, hiringOrganization, jobLocation, employmentType and sometimes baseSalary without any NLP.
2. Isolate the description. Use site-specific CSS selectors where you have them, a readability-style boilerplate remover where you don't.
3. Convert to clean text or Markdown. Keep list structure - bullet lists are where requirements and skills live, and section headings ("Your profile", "What we offer") are strong features for later extraction.

import json
from bs4 import BeautifulSoup

def parse_posting(html: str) -> dict:
    soup = BeautifulSoup(html, "html.parser")

    # 1. schema.org JobPosting from JSON-LD, if the page has one
    structured = None
    for tag in soup.find_all("script", type="application/ld+json"):
        try:
            data = json.loads(tag.string or "")
        except json.JSONDecodeError:
            continue
        items = data if isinstance(data, list) else [data]
        structured = next((d for d in items if d.get("@type") == "JobPosting"), structured)

    # 2. strip non-content elements, keep line breaks for lists
    for tag in soup(["script", "style", "nav", "header", "footer", "form"]):
        tag.decompose()
    text = soup.get_text("\n", strip=True)

    return {"jsonld": structured, "text": text}

Techmap's feed files contain the original html, the original json found on the page (including schema.org data when available) and the extracted text for every posting, and the API returns the description as Markdown in job.jsonLD.description. So you can start from clean text but still go back to the raw page when your parser needs to improve.

3. Normalization

Raw values are inconsistent across sources. Normalization maps them onto shared reference lists so you can group and compare.

Job titles to occupations

"Sr. Java Dev (m/w/d) - Remote" and "Senior Software Engineer, Backend" may describe the same occupation. A typical approach:

1. Clean the title: remove gender markers ("m/f/d"), locations, salary snippets, work-arrangement words and seniority words (store the latter separately).
2. Map to a taxonomy: ISCO-08 (international, 4-digit unit groups), O*NET-SOC (US, with detailed task and skill data), or ESCO (EU, multilingual, linked to ISCO and to a skills pillar). Mapping methods range from exact and fuzzy matching against the taxonomy's alternative labels to embedding similarity (encode the cleaned title and the occupation labels, take the nearest neighbor) and supervised classifiers trained on labeled titles.
3. Keep the confidence score and the taxonomy version with every mapping.

ESCO is often the practical choice for multilingual European data because its occupations come with labels in many languages; O*NET is richer for US-specific analysis. Crosswalks between the systems exist but are many-to-many, so pick one as your primary key.

Company names

"SAP SE", "SAP", "SAP Deutschland SE & Co. KG" and a staffing agency posting "on behalf of a leading software company" are four different strings. Steps that work:

• Strip legal forms (GmbH, Inc., Ltd.) and normalize case, punctuation and Unicode.
• Use the company website domain as a strong key when available.
• Resolve against a reference list (CRM, company register, Wikidata) with fuzzy matching and manual review for the long tail.
• Flag recruiters and staffing agencies separately.

Locations

Location strings range from "Berlin" to "Hybrid - Greater Munich Area" to "Remote (EMEA)". Parse them into city, region and country, then geocode to coordinates with a gazetteer such as GeoNames or a geocoding service. Disambiguation needs the country context of the source ("Cambridge" in the UK vs. Massachusetts). Treat "Remote" as its own value rather than forcing a city.

4. Information extraction

This is where the free text turns into fields. The table summarizes the usual techniques:

Skills: taxonomy matching vs. NER and LLMs

Dictionary matching against a taxonomy (ESCO has a multilingual skills pillar) is transparent, fast and yields IDs for free. Recall is limited to the dictionary, and short tokens need context rules: "R" should only count near words like "statistics".

NER models (spaCy, fine-tuned transformers) find unseen skills but need labeled training data and a mapping step to a taxonomy.

LLMs handle messy text with little setup, at the cost of per-document price at scale, latency, occasional hallucinated values and drift across model versions. A common hybrid: dictionaries for known skills, an LLM over a sample to discover new candidates, human review before they enter the dictionary.

For a worked example of dictionary-based extraction for one tool, see how to identify tool signals in job postings.

Salary parsing

Salary strings need parsing into minimum, maximum, currency and period, then normalization to an annual figure for comparison:

import re

PERIOD_FACTOR = {"hour": 2080, "day": 260, "week": 52, "month": 12, "year": 1}
PATTERN = re.compile(
    r"(?P<cur>[$€£])\s?(?P<min>\d[\d,.]*)\s?(?P<k>k)?"
    r"(?:\s?[-–to]+\s?[$€£]?\s?(?P<max>\d[\d,.]*)\s?(?P=k)?)?"
    r"(?:.{0,20}?(?P<period>hour|day|week|month|year|annum))?",
    re.IGNORECASE,
)

def parse_salary(text: str) -> dict | None:
    m = PATTERN.search(text)
    if not m:
        return None
    def num(s):
        value = float(s.replace(",", ""))  # naive: assumes "," as thousands separator
        return value * 1000 if m.group("k") else value
    period = (m.group("period") or "year").lower().replace("annum", "year")
    low = num(m.group("min"))
    high = num(m.group("max")) if m.group("max") else low
    factor = PERIOD_FACTOR[period]
    return {"currency": m.group("cur"), "min_annual": low * factor,
            "max_annual": high * factor, "period": period}

parse_salary("Pay: $45 - $55 per hour")   # min_annual 93600.0, max_annual 114400.0

This is deliberately minimal. Production parsers also need European number formats ("50.000,00 €"), currency codes, 13th/14th monthly salaries and a filter for numbers that are not salaries (team sizes, years of experience).

What Techmap already extracts

If you work with Techmap data, a part of this layer is already done. According to the job metadata reference and the API field reference, each API record includes:

• occupation: the job title reduced to a stem form (e.g. "java programmer") - not a mapping to ISCO, O*NET or ESCO, which you would add yourself.
• skills and industry: stated on or identified in the posting.
• workPlace (remote, hybrid, onsite, ...), populated only when the original posting states it; careerLevel, contractType and workType.
• hasSalary and minSalary, plus salaryCurrency in the JSON-LD; salary data is sparse because many postings don't disclose it (in the feed export, salary.* is filled for roughly 23% of postings).
• language as an ISO 639-1 code and locale.
• Location fields countryCode, city, state, postCode, a geoPoint with latitude and longitude, plus timezone.
• Company and location identifiers from Techmap's central database in the JSON-LD object, and the flags isDirect (career page vs. job board) and isRecruiter.

The feed files additionally carry position.*, salary.*, contact.* and orgTags.* (tags such as skills, benefits or category paths found on the page). See the data feed overview for the export schema and fill rates.

5. Deduplication

The same vacancy appears on the company career page, on several job boards and on aggregators, and gets re-posted every few weeks. Without deduplication, volume-based trends mostly measure how aggressively companies distribute their ads.

Libraries such as datasketch implement MinHash LSH. Use blocking keys to avoid all-pairs comparisons, and keep duplicates with a pointer to the canonical record - the number of channels a job is advertised on is itself a signal.

Techmap flags duplicates rather than removing them: isDuplicate is true when a posting with the same company, city, country and job title was already seen within 3 days, and the data feeds additionally link each duplicate to the earliest identical posting via isDuplicateOf (a feed-only field). The API accepts isDuplicate=false as a filter. If your use case needs a different definition (longer windows, text similarity), you can build it on top of those fields.

6. Classification and clustering

Supervised classification assigns predefined labels: industry, job family, occupation code, or "is this actually a job posting". Linear models on TF-IDF features are a cheap, strong baseline; fine-tuned transformers or LLM zero-shot classification help when labels are many and training data is scarce.

Unsupervised methods find structure you did not define upfront:

• Topic models (LDA, or BERTopic on embeddings) group descriptions into themes such as "cloud infrastructure" or "clinical care", useful for exploring a new market.
• Embeddings of titles or descriptions place similar jobs close together. Cluster them (HDBSCAN, k-means) to discover job families, or use nearest-neighbor search to map new titles onto known occupations.

Pin model, preprocessing and random seeds if clusters feed a monthly report.

7. Knowledge discovery

With clean, normalized, deduplicated records you can ask questions that individual postings cannot answer.

• Co-occurring skills. Count how often skill pairs appear in the same posting and weight them by pointwise mutual information or lift to filter out generic pairs ("communication" co-occurs with everything). The result is a skill graph that shows which technologies form a stack and which skills bridge occupations.
• Emerging skills. Track each skill's share of postings per month and flag terms with strong relative growth from a meaningful base. Terms that appear in no taxonomy yet but grow quickly are candidates for your dictionary.
• Company signals. Hiring in a new city can indicate an office opening (see how to identify new office locations in job postings); a cluster of postings mentioning a specific tool reveals a technology choice; a sudden rise in sales roles hints at a market push.

Two caveats: postings measure advertised demand, not hires, and coverage differs by country, industry and source type - normalize against a stable baseline before calling something a trend. For an end-to-end walkthrough of the analysis side, see the step-by-step job posting data analysis and analyzing job posting data for business insights.

8. Evaluation

Extraction quality is rarely uniform - a skill extractor can be excellent on English IT postings and poor on German healthcare postings. Measure it explicitly:

• Precision and recall per field. Precision: of the skills you extracted, how many are correct? Recall: of the skills actually in the posting, how many did you find? Report both, and F1 if you need a single number.
• Stratified samples for manual review. Draw a random sample per country, language, source and occupation group, have two annotators label it and check their agreement. A few hundred postings per stratum usually show the major error patterns.
• Coverage metrics. Track the fill rate of each field per source over time. A sudden drop usually means a site changed its markup, not that employers stopped mentioning salaries.
• Regression sets. Run a fixed, labeled set of postings on every parser or model change before deployment, and compare aggregates against official statistics or your own history.

9. Best practices for production pipelines

1. Keep the raw HTML and JSON. Postings disappear; with the raw page you can re-apply every parser improvement to the full history.
2. Version your taxonomies. ESCO, O*NET and your own skill dictionary change between releases. Store the version with every mapped value.
3. Version models and prompts. Record model, version, prompt and parameters for every extraction - otherwise a monthly change may just be a model change.
4. Make the pipeline reproducible. Idempotent jobs keyed on posting ID and processing version, pinned dependencies, an orchestrator such as Airflow or Dagster, and separate raw, clean and derived layers. Re-running a day should produce the same output.
5. Flag, don't delete. Duplicates, low-confidence mappings and suspicious salaries get flags so analysts can choose their own thresholds.
6. Handle personal data deliberately. Restrict access to contact names, emails and phone numbers to the use cases that need them.
7. Document field semantics. The creation date on the page and your scrape date are different things - note which one each metric uses.

Build or buy the extraction layer?

Each stage above is a project of its own. A common split: take acquisition, parsing, basic extraction and deduplication from a provider, and invest your own effort where your use case differs - a domain-specific skill taxonomy, an occupation mapping, or the analytics on top. With Techmap, that means starting from historical datasets (from January 2020) or daily data feeds with raw HTML, text and pre-extracted fields, or querying the Job Posting API with filters such as skills, occupation, workPlace and isDuplicate. The job metadata page lists every field with an example value.