REAL DATA / HISTORICAL EXAMPLE / 21 SEP 2026

What a real run looks like.

A few records. The full context. Nothing made up.

What this example shows

Two real experiments from 21 September 2026: 50 calculus questions and 50 decision scenarios, each with three option orders and four API/input configurations. Each experiment contains 600 attempts. No new model calls were made to create this example.

Historical data, not your live score

These measurements came from the earlier local Chrome/Playwright experiment, not the current browser-only runner. The question renderer, network path, hardware and execution protocol differ. Do not combine these numbers with a new Chan Lab run as if they were the same test.

How the excerpt was chosen

All summary values are recomputed from the complete 1,200 attempts and checked against the original verification files. Only 20 attempt records are included here: the first shared question for all four APIs in each experiment, the valid request closest to each API’s median latency, and all four failed requests. This is a deterministic illustration, not a random sample; do not recompute overall scores from these 20 rows.

Setup and timing

Jev responses identified jev-1.13.0. OpenJev used AlexWortega/openjev/qwen3.5-4b-nli-v2 on an Apple M4 Pro with 48 GB unified memory, MPS and BF16. Text, screenshot and combined inputs were compared. The cloud and local deployments used different hardware and network paths. HTTP duration includes network and service processing through response parsing and validation; it is not pure model thinking time. P50/P95 use linear interpolation and include failed attempts.

S2-Q01-R1
For f(x) = 3x³ + 4x² − 5x, find f′(1).

A. 8
B. 2
C. 12
D. 17

Answer key: C

D1-Q08-R1
School poster. One person; hands-on steps cannot overlap or pause. Alone phases can overlap anything.
Design: 2m hands.
Print: 1m hands, then 5m alone; after Design fully finish.
Frame: 4m hands.
Mount: 2m hands; after Print + Frame fully finish.
Tidy: 2m hands.
Follow each listed order, waiting if needed. Which finishes all work earliest?

A. Design → Tidy → Print → Frame → Mount
B. Design → Print → Frame → Mount → Tidy
C. Tidy → Design → Print → Frame → Mount
D. Design → Print → Tidy → Frame → Mount

Answer key: D

Interpretation and limits

The two banks are not pooled into one accuracy or timing score. Repeated option orders are not independent questions. There was no external difficulty calibration. The original evaluator clicked the returned option using Playwright; it did not test independent coordinate grounding. Cloud Jev and local OpenJev latency are not a controlled hardware comparison. The original screenshot sizes also differ from the current browser-rendered cards.

Publication and provenance

Only an explicit field allowlist is exported: API group, question IDs, selected/gold answers, correctness, HTTP status/error, UTC timestamps and stage durations. Credentials, Authorization headers, API endpoint URLs, raw request/response bodies, machine paths and internal document links are excluded. The historical measurements are static public examples; visitors’ new results remain in their browser.

The downloadable JSON preserves exact aggregate values and provenance hashes. The CSV preserves original UTC timestamps and all selected timing stages. Full-run source files are not included in this excerpt. Mermaid source for the four charts is in the Markdown download.

Try your own APIs →