Chan Lab

REAL DATA / HISTORICAL EXAMPLE / 21 SEP 2026

What a real run looks like.

A few records. The full context. Nothing made up.

Markdown + Mermaid ↓Summary & sample JSON ↓24 selected records · CSV ↓

What this example shows

Two real experiments from 21 September 2026: 50 calculus questions and 50 decision scenarios, each with three option orders and five API/input configurations. Each experiment contains 750 attempts. Laya used the same questions in a later measurement on that date; it was not run concurrently with the other groups.

Historical data, not your live score

These measurements came from the earlier local Chrome/Playwright experiment, not the current browser-only runner. The question renderer, network path, hardware and execution protocol differ. Do not combine these numbers with a new Chan Lab run as if they were the same test.

How the excerpt was chosen

All summary values are recomputed from the complete 1,500 attempts and checked against the original verification files. Only 24 attempt records are included here: the first shared question for all five configurations in each experiment, the valid request closest to each API’s median latency, and all four failed requests. This is a deterministic illustration, not a random sample; do not recompute overall scores from these 24 rows.

Setup and timing

Jev responses identified jev-1.13.0. OpenJev used AlexWortega/openjev/qwen3.5-4b-nli-v2 on an Apple M4 Pro with 48 GB unified memory, MPS and BF16. Text, screenshot and combined inputs were compared. The cloud and local deployments used different hardware and network paths. HTTP duration includes network and service processing through response parsing and validation; it is not pure model thinking time. P50/P95 use linear interpolation and include failed attempts. Laya used 421M English weights on MLX Metal FP16, without compilation or prompt caching, on the same Mac.

Calculus — five sets

50 base questions × 3 rounds × 5 groups = 750 attempts. Each API’s fixed denominator is 150, including failures.

API / inputCorrect / 150AccuracyValid accuracyFailuresHTTP mean sP50 sP95 s
Jev text127/15084.7%84.7% (127/150)00.8630.6681.393
OpenJev text79/15052.7%52.7% (79/150)01.5971.6011.713
OpenJev image72/15048.0%48.0% (72/150)04.1634.2294.362
OpenJev text + image83/15055.3%55.3% (83/150)04.4124.4824.616
Laya text36/15024.0%24.0% (36/150)00.0390.0380.043
Calculus — five sets accuracyCalculus — five sets accuracyJev text84.67 %OpenJev text52.67 %OpenJev image48.00 %OpenJev text + image55.33 %Laya text24.00 % Calculus — five sets mean API timeCalculus — five sets mean API timeJev text0.86 sOpenJev text1.60 sOpenJev image4.16 sOpenJev text + image4.41 sLaya text0.04 s

Per-round data

API / inputRound 1 / 50Round 2 / 50Round 3 / 50
Jev text44/5040/5043/50
OpenJev text27/5024/5028/50
OpenJev image24/5024/5024/50
OpenJev text + image28/5026/5029/50
Laya text14/5012/5010/50

A question shared by all five configurations

S2-Q01-R1
For f(x) = 3x³ + 4x² − 5x, find f′(1).

A. 8
B. 2
C. 12
D. 17

Answer key: C

Selected attempt records

Group #API / inputQuestionChoice / goldResultHTTP secondsWhy included
1Jev textS2-Q01-R1D / Cincorrect3.077755first shared question
1Laya textS2-Q01-R1A / Cincorrect0.173281first shared question
2OpenJev textS2-Q01-R1A / Cincorrect1.816309first shared question
3OpenJev imageS2-Q01-R1A / Cincorrect4.252545first shared question
4OpenJev text + imageS2-Q01-R1A / Cincorrect4.613546first shared question
59Laya textS4-Q05-R2A / Dincorrect0.038060nearest valid latency median
138OpenJev text + imageS1-Q04-R1D / Dcorrect4.484471nearest valid latency median
158Jev textS4-Q04-R1B / Bcorrect0.668401nearest valid latency median
172OpenJev textS5-Q05-R1B / Aincorrect1.602863nearest valid latency median
198OpenJev imageS5-Q07-R1C / Ccorrect4.228928nearest valid latency median

In this calculus bank, Jev text scored 127/150 = 84.7%; OpenJev text, image and combined input scored 52.7%, 48.0% and 55.3%. Laya text scored 36/150 = 24.0%, with mean HTTP time 39.31 ms. No requests failed. These observed scores describe this fixed question bank and deployment, not a general ranking.

Decision scenarios — five sets

50 base questions × 3 rounds × 5 groups = 750 attempts. Each API’s fixed denominator is 150, including failures.

API / inputCorrect / 150AccuracyValid accuracyFailuresHTTP mean sP50 sP95 s
Jev text122/15081.3%83.6% (122/146)41.1480.7252.194
OpenJev text93/15062.0%62.0% (93/150)02.3322.2042.888
OpenJev image90/15060.0%60.0% (90/150)04.8154.7804.963
OpenJev text + image105/15070.0%70.0% (105/150)05.8755.7356.422
Laya text43/15028.7%28.7% (43/150)00.0450.0440.057
Decision scenarios — five sets accuracyDecision scenarios — five sets accuracyJev text81.33 %OpenJev text62.00 %OpenJev image60.00 %OpenJev text + image70.00 %Laya text28.67 % Decision scenarios — five sets mean API timeDecision scenarios — five sets mean API timeJev text1.15 sOpenJev text2.33 sOpenJev image4.82 sOpenJev text + image5.87 sLaya text0.04 s

Per-round data

API / inputRound 1 / 50Round 2 / 50Round 3 / 50
Jev text39/5043/5040/50
OpenJev text32/5030/5031/50
OpenJev image30/5032/5028/50
OpenJev text + image37/5035/5033/50
Laya text10/5014/5019/50

A question shared by all five configurations

D1-Q08-R1
School poster. One person; hands-on steps cannot overlap or pause. Alone phases can overlap anything.
Design: 2m hands.
Print: 1m hands, then 5m alone; after Design fully finish.
Frame: 4m hands.
Mount: 2m hands; after Print + Frame fully finish.
Tidy: 2m hands.
Follow each listed order, waiting if needed. Which finishes all work earliest?

A. Design → Tidy → Print → Frame → Mount
B. Design → Print → Frame → Mount → Tidy
C. Tidy → Design → Print → Frame → Mount
D. Design → Print → Tidy → Frame → Mount

Answer key: D

Selected attempt records

Group #API / inputQuestionChoice / goldResultHTTP secondsWhy included
1Jev textD1-Q08-R1B / Dincorrect0.937435first shared question
1Laya textD1-Q08-R1C / Dincorrect0.064736first shared question
2OpenJev textD1-Q08-R1B / Dincorrect3.942378first shared question
3OpenJev imageD1-Q08-R1B / Dincorrect5.378662first shared question
4OpenJev text + imageD1-Q08-R1B / Dincorrect7.111950first shared question
78Jev textD2-Q03-R1— / Afetch failed [UND_ERR_CONNECT_TIMEOUT]10.380249all failed attempts
87Laya textD5-Q08-R2D / Aincorrect0.043516nearest valid latency median
108OpenJev textD2-Q02-R1D / Dcorrect2.203423nearest valid latency median
150OpenJev imageD4-Q10-R1C / Dincorrect4.780264nearest valid latency median
152Jev textD4-Q10-R1D / Dcorrect0.716782nearest valid latency median
263OpenJev text + imageD5-Q09-R2B / Bcorrect5.736382nearest valid latency median
568Jev textD2-Q06-R3— / Bfetch failed [UND_ERR_CONNECT_TIMEOUT]10.361398all failed attempts
587Jev textD1-Q08-R3— / Bfetch failed [UND_ERR_CONNECT_TIMEOUT]10.368178all failed attempts
600Jev textD3-Q02-R3— / Dfetch failed [UND_ERR_CONNECT_TIMEOUT]10.385349all failed attempts

Four Jev calls ended in connection timeouts (UND_ERR_CONNECT_TIMEOUT), with no HTTP response. They count as unsuccessful attempts in 122/150 = 81.3%; valid-answer accuracy is 122/146 = 83.6%. These failures do not prove a reasoning error. No retries were substituted. OpenJev combined input scored 105/150 = 70.0%, with a mean HTTP duration of 5.875 seconds in this local deployment. Laya text scored 43/150 = 28.7%, with mean HTTP time 44.87 ms.

2048 — same first 32 moves

Laya was measured on 22 September 2026 with the original Jev seed, structured state, instructions and direction descriptions. The table compares the first 32 moves with the earlier Jev/OpenJev trajectories. Later boards differ because actions differ. These are deployment observations from single trajectories, not a win-rate ranking.

API / inputMovesScoreLargest tileHTTP mean msHTTP P50 msHTTP P95 ms
Jev text3212416363.88292.50877.60
OpenJev text32128161524.461613.411687.43
OpenJev image32124164007.214332.244809.34
OpenJev text + image32200324651.944935.275397.12
Laya text321883252.2541.2562.04
2048 score at move 322048 score at move 32Jev text124.00 OpenJev text128.00 OpenJev image124.00 OpenJev text + image200.00 Laya text188.00 2048 mean API time at 32 moves2048 mean API time at 32 movesJev text0.36 sOpenJev text1.52 sOpenJev image4.01 sOpenJev text + image4.65 sLaya text0.05 s

Laya ended naturally after 144 moves: score 1,400, largest tile 128. No requests failed. Mean HTTP time was 43.30 ms (P50 40.67, P95 43.28); mean inference time was 40.15 ms; mean complete-step time was 120.99 ms. All requests, including the slower first game request, remain in these statistics. Model loading and service warmups are separate.

The Jev historical run stopped at its API budget after 195 moves (score 2,348, tile 256); all OpenJev modes stopped at 32 moves. These endpoints are not equivalent. Laya is text-only; its 289 images document the run. Independent replay, image hashes, UTC ordering and the actual tokenizer confirmed all 144 steps and no truncated instructions, options or state.

2048 JSON 144-step timing CSV

Interpretation and limits

The two banks are not pooled into one accuracy or timing score. Repeated option orders are not independent questions. There was no external difficulty calibration. The original evaluator clicked the returned option using Playwright; it did not test independent coordinate grounding. Cloud Jev, local OpenJev and local Laya latency are not a controlled hardware comparison. The original screenshot sizes also differ from the current browser-rendered cards.

Publication and provenance

Only an explicit field allowlist is exported: API group, question IDs, selected/gold answers, correctness, HTTP status/error, UTC timestamps and stage durations. Credentials, Authorization headers, API endpoint URLs, raw request/response bodies, machine paths and internal document links are excluded. The historical measurements are static public examples; visitors’ new results remain in their browser.

The downloadable JSON preserves exact aggregate values and provenance hashes. The CSV preserves original UTC timestamps and all selected timing stages. Full-run source files are not included in this excerpt. Mermaid source for the six charts is in the Markdown download.

Try your own APIs →