AI Benchmarks Overlook 90% of U.S. Working Hours, Mercor Research Finds

Artificial intelligence benchmarks currently leave the vast majority of work in the United States unmeasured, according to research published by evaluation platform Mercor and discussed on Hacker News (https://www.mercor.com/research/labor-index/benchmark-map/). The company's Benchmark Map indicates that 90% of American working hours are spent on tasks with little or no coverage from existing evaluation suites.

To evaluate how current tests align with real-world employment, Mercor mapped 29 economically relevant AI benchmarks against the U.S. Department of Labor's O*NET taxonomy. Across the database's 923 occupations and 18,797 categorized tasks, only 208 occupations contain a single task evaluated by at least five benchmark prompts. Six of the 22 broad O*NET job families have no occupations meeting that threshold.

The researchers found that evaluation datasets heavily concentrate on high-wage positions deemed susceptible to automation. While prompts frequently test tasks performed by corporate lawyers, financial analysts, and software engineers, they rarely evaluate tasks belonging to adjacent roles like legal secretaries or court clerks.

Statistical modeling in the report indicates that tasks tagged as exposed to AI automation are roughly six times more likely to appear in benchmarks than unexposed tasks. Within occupations, tasks rated by domain experts as most critical are six to ten times more likely to be covered than low-importance tasks. Benchmark representation also correlates strongly with total occupational wage bill, even after controlling for AI exposure.

Mercor's methodology used large language model judges to map benchmark prompts to O*NET tasks, validating the automated tags against human expert ratings on its proprietary APEX benchmark suite. An O*NET task was classified as covered if mapped to at least five prompts. Mercor noted its findings are provisional due to benchmark selection constraints and the exclusion of physical labor, citing research from Dingel and Neiman estimating that 63% of U.S. jobs involve tasks with physical-world components.

To assess whether weighted scoring could balance evaluation, Mercor tested task weighting across four professions: investment banking, law, accounting, and software engineering. While weighted and unweighted model scores remained similar, researchers observed that less than one-third of the relevant O*NET tasks in those fields currently have sufficient prompt volume, highlighting evaluation gaps for enterprise buyers and economic forecasters.