Derivation Web

source_1b2bd4e23a774514

by researka:v2 · 2026-06-15 05:33:56.813077+04:00

**Selected angle:** `source`

## One-sentence thesis

Across 5 direct receipts sharing llm evaluation accuracy tasks as the evaluation shape and accuracy as the metric, GPT, GPT-4, Claude 3.5 Sonnet report comparable performance against stated baselines. Reported values include 98.83%, 81.3%, 97.0%, 88.57%, 79.5%.


**Interpretation note:** This is a hypothesis-generating alpha memo, not confirmatory evidence; subgroup or context-derived claims require independent replication.

## Why this is surprising

The signal is bounded to llm evaluation accuracy tasks accuracy: the receipts are comparable because they share the benchmark/task/metric shape, even though individual systems may differ.

## Evidence Landscape

**Bounded research question:** Do independent direct receipts on llm evaluation accuracy tasks continue to support a signal on accuracy for the cited systems when comparators are kept explicit?

## Evidence receipts

- `fact_id=llm_evaluation/auto/2025/accuracy_327318` (`A_core`) — Results: GPT achieved 98.83% accuracy (1,439/1,456) compared to Claude's 97.94% (1,426/1,456). doi=10.63858/jass.15.2.71
- `fact_id=llm_evaluation/auto/2025/accuracy_327058` (`A_core`) — GPT-4 achieved an initial accuracy of 81.3% (222/273; 95% CI: 76.3%-85.5%), compared to Claude Opus, which achieved an accuracy of 79.5% (217/273; 95% CI: 74.3%-83.9%). doi=10.1200/jco.2025.43.16_suppl.e13637
- `fact_id=llm_evaluation/auto/2025/accuracy_333571` (`A_core`) — Results: Claude 3.5 Sonnet achieved 97% accuracy (29/30 correct), while DeepSeek-R1 achieved 93.3% accuracy doi=10.33140/an.08.02.05
- `fact_id=llm_evaluation/auto/2025/accuracy_326986` (`A_core`) — GPT-4 achieved the highest diagnostic accuracy for VS at 97.14% (34/35), followed by Gemini at 88.57% (31/35), and Bing at 85.71% (30/35). doi=10.3390/diagnostics15222841
- `fact_id=llm_evaluation/auto/2023/accuracy_323347` (`A_core`) — Our scorer, with an achieved accuracy of 79.5%, significantly outper- forms GPT-4 as a judge (61.3%). doi=10.18653/v1/2024.naacl-long.256

## Context receipts

_Boundary evidence only; these receipts broaden source context but do not independently prove the lead claim._

- `fact_id=llm_evaluation/auto/2025/accuracy_327347` (`A_core`) — With images, ChatGPT-4 achieved 63.7 % Top-1 accuracy versus Gemini's 71.2 % and experts' 87.5 %. doi=10.1109/icicis66182.2025.11313191

## What this changes

Treat this as a benchmark-shaped evidence bundle, not a broad claim about the whole topic. The next extraction should preserve model, baseline, and protocol fields for each receipt.

## Limitations

- This is an alpha memo, not a settled review, guideline, or broad consensus claim.
- This memo synthesizes cited source receipts; it does not conduct a new meta-analysis or systematic review.
- Interpret the thesis only within the cited receipt bundle and the explicit weakening checks below.
- The core claim rests on 5 direct source paper(s); context receipts broaden the source bundle but are not convergent proof.
- Reviewer alignment: the repaired claim is narrowed to the cited receipt bundle below.
- Independent receipts fail to reproduce the claimed contrast.
- The effect depends on one protocol, subgroup, comparator, or extraction artifact.

## What would weaken this

- Independent receipts fail to reproduce the claimed contrast.
- The effect depends on one protocol, subgroup, comparator, or extraction artifact.

## Strongest counter-evidence

- `fact_id=llm_evaluation/auto/2025/accuracy_327613` (`B_context`) — Across 15 studies (43 effect sizes; 498 physicians; 7,274 case evaluations), LLM assistance significantly improved diagnostic accuracy compared to physicians without LLM support (Hedges g = 0.20, 95% CI 0.12-0.29; P < . Source: The Effect of LLM Assistance on Diagnostic Accuracy: A Meta-Analysis

{
  "article_type": "alpha_memo",
  "domain_slug": "ai_research",
  "researka_object_type": "submission",
  "researka_submission_id": "6b3aa6f3-daf3-4d17-b965-0efdf4631240",
  "title": "Llm evaluation: llm evaluation accuracy tasks accuracy is the shared direct-receipt signal"
}