AI tests cover a fraction of American work. Companies need their own trials.

A business owner choosing an AI assistant can compare scores for coding, mathematics and professional exams. A benchmark is a standardized test used to compare AI systems. Whether a high score predicts faster service, fewer mistakes or lower costs inside a particular company requires a different kind of evidence.

Mercor’s Benchmark Map matched 29 selected tests to American occupations. In its snapshot, 208 of 923 occupations had at least one task represented by five or more test prompts. Across individual tasks, coverage reached 924 of 18,797, about 4.9%. Its estimate that roughly 90% of working hours receive little or no coverage uses modeled time shares. These figures describe its selected index, rather than every available AI test.

For owners, the practical buying decision concerns a narrower unit: a task someone must complete using company systems, under company rules. Published research shows that AI can help one group of employees while slowing another. Even within a single profession, assistance can improve some assignments and make others less accurate.

How a job becomes a test

O*NET, a U.S. Department of Labor occupational database, describes jobs through tasks, skills and working conditions. Its current release is version 31.0; Mercor used version 30.0. A national job description provides a common vocabulary for researchers. A company still needs to identify its own software, procedures and approval requirements before using that description to choose an AI system.

Mercor’s selection rules favor tests that ask AI to carry out economically relevant assignments. They also limit representation from heavily tested fields and exclude tests already mastered by leading models. Those choices broaden the index but can leave out useful, easier tasks. Adding benchmarks or changing selection rules could change measured coverage.

Mapping also involves judgment. Mercor’s methodology uses an AI model to connect prompts with occupational tasks. Experts reviewed primary assignments for 1,013 prompts from three APEX benchmarks, accepting the first choice in 76.7% of 1,763 judgments. Some prompts had multiple reviewers. The map includes additional candidate assignments that experts did not all review. A mapped task therefore identifies a proposed relationship between an exercise and a job activity; it does not report how well a model performed that activity.

Workplace studies measure different outcomes

In customer support, Erik Brynjolfsson, Danielle Li and Lindsey Raymond examined a gradual rollout of an assistant at one business software company. Their 2025 Quarterly Journal of Economics paper, covering 5,172 agents, found that access increased successfully resolved chats per hour by 15% on average. Less experienced and lower skilled employees gained more. Agents could accept, edit or ignore suggested replies, so the finding concerns people using assistance.

Consultants encountered a more uneven pattern. A study of 758 Boston Consulting Group consultants, published in Organization Science in March 2026, tested the June 2023 version of GPT-4 on designed assignments. On 18 tasks suited to its capabilities, participants with access completed 12.2% more tasks and used 25.1% less time. On one separate problem outside those capabilities, AI users were 19 percentage points less likely to reach the correct solution.

METR’s randomized software study supplied another contrast. Sixteen experienced open-source developers completed 246 tasks in repositories they knew. With February–June 2025 tools available, they took 19% longer. Developers had expected assistance to speed them up. In a February 2026 follow-up, METR said newer tools probably helped more, but changes in who participated and which tasks they submitted prevented a reliable updated estimate.

A company comparing vendors should ask which study outcome resembles its own decision, which employees participated and whether current tools differ from those studied. A result from experienced maintainers using familiar repositories answers a different buying question from a result about new support agents learning to respond to customers.

Build a trial around a complete assignment

The National Institute of Standards and Technology’s AI evaluation guidance calls for testing under conditions similar to deployment, documenting methods and involving people with relevant expertise. It also recommends evaluating performance after deployment. For a company, that supports a trial using actual assignments, the information employees normally receive and clear rules for human review.

SnowRock proposes starting with one repeatable assignment and a written definition of an acceptable result. Select cases from normal operations, including exceptions employees regularly handle. Preserve what was known when each assignment began. Later corrections and final answers belong in the scoring material, separate from information supplied to the model.

Compare similar cases with and without assistance, and assign them randomly where practical. Keep employee experience and task difficulty in view. Reviewers should apply the same standards to both groups and, when possible, assess outputs without knowing which method produced them. Count time spent reading suggestions, correcting answers and completing required approvals, alongside drafting time.

Before beginning, agree which errors would make the tool unsuitable. An incorrect internal summary and an incorrect payment instruction can have very different costs. Report serious mistakes separately so that faster routine cases cannot hide them in an average. Track how often an employee must take over and whether customers reopen supposedly completed requests.

Decide how results will be grouped before seeing them. Routine assignments can dominate a test and conceal weak performance on uncommon but important cases. Report those categories separately. Compare experienced employees with newer staff as well, because assistance may change what each group can accomplish. Keep abandoned attempts in the record rather than counting only assignments that finish successfully.

A small trial can expose obvious failures without settling whether a modest improvement will last. Report how many cases were completed and how widely completion times varied. If results depend on one employee or a few unusually easy assignments, extend testing before treating that improvement as a budget assumption.

Add software fees, employee time, setup and review expenses, then divide by assignments that meet the agreed standard. Compare that figure with the existing process. A cheaper subscription can still require expensive supervision; a higher priced tool can be economical if it reduces total effort while meeting quality requirements.

For companies considering data licensing, the support study supplies a concrete example: its assistant used earlier customer-agent conversations and internal documentation. A buyer seeking similar records should identify which tasks, supporting documents and outcomes an evaluation requires. The owner should agree what can be shared, which uses are permitted and who pays for preparation. A missing public benchmark alone provides no evidence that a particular archive has a buyer.

Record the model version, instructions, permitted tools and test dates so later comparisons remain interpretable. Keep a portion of cases unseen during adjustments, and repeat checks when the software or company process changes. Before expanding a purchase, the owner should have counts of acceptable assignments, serious errors, employee hours and total spending from the company’s own trial.