Portfolio case study 001
AI Benchmarks and Reliable Professional Performance
How far does a benchmark result travel?
No model vendor commissioned or reviewed this work.
Question
Do strong benchmark scores predict dependable professional work?
The review examines the distance between performance on a controlled test and acceptable, repeated performance in a working environment. It separates comparison between systems from prediction of workplace outcomes.
Finding
A useful signal, with a defined scope.
Benchmarks can help compare systems on closely matched, bounded tasks. The paraphrase study, for example, found relatively stable rankings despite lower absolute scores; that does not establish workplace prediction. A score alone does not establish an operational failure rate, productivity gain, or readiness for autonomous professional use. The reviewed studies use different tasks, tools, outcomes, and levels of supervision; their percentages should not be pooled into a single estimate.
This review did not locate a broad prospective study connecting standardized benchmark scores to subsequent workplace outcomes for the same systems across multiple professional settings. That is a bounded search finding, not proof that no such study exists.
Evidence
What the source trail shows
Professional-task evaluations such as GDPval, SWE-Lancer, and the Remote Labor Index measure different portions of work. Their findings require the task scope and scoring standard to travel with the result.
Repeated-trial studies expose a separate question: does a system keep succeeding? In the audited tau-bench version, GPT-4o’s retail single-trial success (pass1) was 61.2%, while success across all eight trials (pass8) was about 25%. The revised reliability study covers 15 models and reports limited overall reliability gains relative to capability gains within its two benchmark settings.
Evidence of practical value also matters. Studies of assisted programming and customer support report gains in their settings. These findings counter a blanket claim that AI assistance has no utility; they do not establish that benchmark scores predicted those gains.
Method
Keep the claim, evidence, and inference together.
The companion workbook records 25 claims, 22 source-map entries, 14 verification notes, and 10 unresolved questions. The narrative cites S01–S20; the remaining two entries are discovery and context links. Verification fields identify source locations and distinguish published findings from interpretation and unperformed checks.
The review considers evidence against its emerging conclusion, records version-sensitive corrections, and identifies what additional work could change the answer. It does not independently reproduce the experiments or assess transfer to a specific client’s operating environment.
Diagnostic Research proposes examining task, success, system, workflow, reliability, and time when considering transfer. These dimensions guide local evaluation; they are not a validated certification scheme or universal deployment threshold.
Read the research methodResearch package
Read the reasoning. Inspect the records.
Corrected edition, 10 September 2026. Open evidence gaps remain identified in the documents. Source checking is not experimental replication. Client-specific transfer has not been assessed.
Related reading: AI Benchmark Evaluation, a separate five-source research sample.