Methodology and caveats

What this data is, how it is built, and the ways it can mislead.

Sources

Public job-board APIs of three applicant tracking systems. No scraping of HTML, no logins, no aggregators.

Greenhouse boards-api.greenhouse.io/v1/boards/<board>/jobs
Ashby api.ashbyhq.com/posting-api/job-board/<board>
Lever api.lever.co/v0/postings/<board>

The board list (data/boards.csv) holds 2,898 boards: 1,506 ashby, 1,095 greenhouse, 297 lever. Requests are paced to one start every 0.3 seconds per API host with a descriptive User-Agent, once a day.

Coverage bias

Startup-heavy. The board list was assembled from venture-capital portfolio job boards, accelerator directories and public hiring threads, so it over-represents venture-backed tech and scale-ups.
Big enterprises are missing. Banks, retailers, insurers, government and most of the Fortune 500 use Workday, iCIMS, Taleo or SuccessFactors, which are not crawled. So are staffing agencies. Read every number as "the startup and scale-up market", not "the US market".
Board churn. A company that switches ATS disappears until its new board is added.

Pipeline

Ingest (Python): every board is fetched; postings whose title matches a data-role pattern are landed as Parquet, one file per board per day, with the raw ATS JSON kept verbatim (bronze). Runs are resumable and idempotent.
History: each day is folded into four small Parquet files: postings with first and last sighting, daily sightings, board-crawl results, and SCD type 2 versions of each posting (title, location, pay range, description hash).
Warehouse (DuckDB + dbt): staging, intermediate and mart models; tests on every key, relationship and enumerated column; a hand-labelled title set that the classifier must reproduce exactly. Browse the dbt docs and lineage graph.
Dashboard: static pages rebuilt from the marts after every run.

Definitions

Role family and seniority come from the title alone, using ordered regex tables in seed CSVs. Sales, recruiting, product-management and other non-data titles are excluded even when they contain "data" or "ML".
Remote-US needs a remote signal (a location that says remote, or an ATS workplace type of Remote) and a location that is in the US, is North America, or names no country. "Remote" with no country counts as US-compatible, which over-counts some roles that are remote within another country.
Listing = one company plus one normalised title. The same role posted in five cities counts once.
Open / closed: a posting closes on the first day its board is crawled successfully and the posting is gone. A failed crawl never closes anything. Lifespans of open postings are right-censored.
Doorway = an intern, entry, mid or unlabeled title that asks for five years of experience or less (or does not say).

Keyword matching counts mentions, not requirements

Skills are 83 documented patterns (skill_taxonomy.csv) matched against the description. "Nice to have: Spark", "our stack includes Spark" and "you will never touch Spark" all count as a mention. Company boilerplate (benefits, values, "we use AI to...") also produces mentions, which inflates generic terms such as communication, LLM and dashboards. Shares are therefore upper bounds on how often a skill is actually required.

Pay

The ATS's structured pay field is used first, in any currency and interval, converted with the fixed rates in fx_rates.csv and annualised (hour x 2,080, month x 12).
Otherwise the description is searched for "$X - $Y" ranges. When several ranges are listed (geographic zones), the highest floor is kept, which is usually the tier-1 US zone, so text-parsed pay leans high.
Ranges outside $30k to $900k a year, or whose top is three or more times the bottom, are treated as unparseable.
The midpoint of a range is not what people are paid. Posted ranges are often wide and can include equity-rich or bonus-heavy packages that are not visible here.

Degrees and years

A degree is "named" when the word appears anywhere, including "or equivalent experience" and "preferred". Years asked is the highest "N years ... experience" phrase, which is a proxy for the bar, not a hard cut-off.