Chan Lab

BROWSER EVALUATION · 2026-09-22

Know what you are measuring.

Chan Lab evaluates multiple-choice answers, reproducible solo games and two-model chess matches. Your browser runs the environment, records and scores responses, and generates Markdown reports with Mermaid charts and measured data. Model inference runs at your selected endpoint. These tests do not measure mouse-coordinate grounding, independent web navigation or executing real-world tasks.

Your key and your data

Dynamic question bank and replay

Questions are maintained in the public GitHub question bank. The browser loads its versioned JSON catalog on page load or when you click Refresh question bank, then checks the dataset schema and SHA-256. Compatible bank updates do not require a website release. A running evaluation keeps its loaded version. If the remote catalog is unavailable, the site clearly labels its bundled snapshot; hash or dataset validation failures block evaluation.

Each run draws a new sample without replacement, balanced across sets and topics. All APIs in the batch receive identical question IDs and option orders. A random seed, catalog and dataset hashes, source URLs and exact question IDs are exported in JSON and Markdown. Enter a seed under Advanced to reproduce a sample with the same bank version and settings. Short and Medium are nested only when their seeds match. Long is capped at 50 base questions so future growth cannot silently increase costs.

Question-bank downloads are public GET requests with no API key or cookies. Model keys go only to the selected model endpoint. The manifest and dataset contract is ready for another HTTP backend.

Read a real example

View the historical example report: verified full-run aggregates from 1,500 earlier attempts, plus 24 explicitly selected records. Download its Markdown/Mermaid source, JSON and CSV. It is an owner-provided public example from the earlier Mac/Playwright experiment, not a score from the current browser runner or a visitor’s uploaded result.

How to run a comparison

  1. Choose API format, Base URL, model, key and input mode. Accept your provider’s usage charges. For one API, run directly.
  2. For multiple APIs, click Add this API to comparison for each connection. Different models or modes may share a provider; keys remain associated with their own connection. Up to eight configurations per batch.
  3. Select a shared dataset, set and run length: Short uses 5 base questions, Medium 15, and Long up to 50 in a large bank. For a 10-question set or pilot, the sizes are 3 / 6 / 10. The default is one round. Advanced options can repeat the same questions with three fixed option orders, multiplying calls by three. The total request budget includes every API.
  4. Keep the tab open. Requests run sequentially; the first API rotates on each question. Each request has one attempt, with no retries and no previous answers fed back.
  5. Download Report .md for a complete Markdown report, data tables and Mermaid charts. JSON and CSV contain exact timings. Evidence ZIP adds the report and per-question PNGs. No extra model calls write the report.

Datasets and answer keys

The 10-question calculus pilot is separate from the 50-question expanded calculus bank. The expanded bank uses 10 topic templates, each with five parameterizations. The 50 decision scenarios cover parallel scheduling, prerequisites and resources, logical constraints, ordering, and plan changes. Prompts are English; answer explanations retain the original Chinese notes. We preserve the original question text, options and answers.

The two larger banks have three predetermined option permutations per question. These are 50 base questions, not 150 independent questions. Each JSON download contains the public answer key and source hashes. The evaluator hashes the dataset against the published manifest before running. Calculus answers were symbolically checked; scenario options were checked by enumeration and an independent state/schedule validator in the original experiment. This does not establish external difficulty calibration.

Answers and explanations never enter the model request. They remain available to anyone downloading the dataset, so this is a reproducible self-evaluation tool, not a cheating-resistant leaderboard. For scheduling questions, all listed schedules may be legal, while only one finishes earliest. An incorrect choice may simply be suboptimal.

Multiple-choice inputs and API adapters

Image cards are browser-rendered canvases, 900 pixels wide with enough height for the complete question, using a 22-pixel body font. The exact ungraded PNG is sent to each image-enabled API. Result cards are drawn after scoring. They are not operating-system screenshots or evidence of a trusted mouse click. This rendering and the browser request environment differ from the original Mac/Playwright runs: do not merge the two experiments into one ranking.

Short, medium and long

Short and Medium use a deterministic, nested sample: first balance selected sets, then topics, with base ID as a stable tie-breaker. They do not simply take the first few questions. All APIs receive the same question IDs; the selected base questions repeat across option rounds. Reports record the preset, sampling version and complete question plan. A short check is not a full-dataset score.

Before measurements exist, the time estimate assumes 1–6 seconds per request and multiplies by the total calls across APIs. This is a planning range, not a speed guarantee. During a run, remaining time is estimated from measured per-API trial durations; subsequent plans can use matching measurements in this tab. There are no extra probe requests. Font and color theme preferences are held only in page memory.

Scoring and time definitions

Local browser connector

Install Chan Lab Connector before running. Every run checks extension availability and endpoint permission before sending credentials. Jev rejects direct web-page CORS requests; the connector instead uses authorized extension background requests. Model inference stays at the selected API; all scoring and reports stay in this tab.

Jev permission is included; other HTTPS API origins require explicit permission in the extension. No automatic cloud fallback. Connection, authentication and timeout failures remain visible and stop affected requests. End-to-end API timings include extension messaging overhead.

Inspect it yourself

Download the static site source or inspect API adapters, browser runner, report generator and dataset hashes. In DevTools → Network, model calls should target only your chosen endpoint; no Key or model result is sent to this website. The ZIP library is vendored locally with its license.

Model arena: board games

Open the model arena →. All seats are models. Add two model-seat configurations; the same model may occupy both with independent settings. The seats alternate on one shared board; each request uses the active model seat's endpoint, key and input mode. A rules engine supplies all legal moves in a stable order, without a chess search engine, move ratings or candidate filtering. Waiting for an API changes neither the board nor a chess clock.

Short schedules 2 matches of at most 32 plies each; Medium 6 matches of 128 plies; Long 10 matches of 256 plies. One ply is one model seat's action. Every pair of matches swaps the model seats. All use the standard starting board: the seed identifies rotation groups and does not randomize openings. More games from the same start are not independent opening samples.

The rules version is chess-js-1.4-auto-draw-v1. Checkmate, stalemate, castling, en passant and all four promotions follow chess.js 1.4.0. Threefold repetition, the 50-move rule (100 half-moves without a pawn move or capture), and insufficient material detected by the library end automatically in a draw. This is the site's automatic-draw variant, not the complete FIDE claim procedure. There is no pass or resignation action.

Other supported rules

Rules and results are versioned by game. Do not pool outcomes across these different games or scoring variants. The rule engine never selects a move for strategic strength. Chess uses UCI move IDs; other games retain their own coordinate IDs and exact histories.

Action selection and request budgets

Model inputs, results and evidence

Text sends the assigned seat, full current public observation, rules and legal candidate IDs with neutral move descriptions. Text + image adds the same position's PNG. Image-only text contains rules, candidate IDs and labels, and minimal game/seat/turn identifiers; it contains no FEN, board layout or move history that could reconstruct the board. The PNG includes the complete board, special-move rights, counters and full move history, with height extended instead of truncating history. Jev is text-only. Image and text payloads are different representations, not an equal-token control.

Only naturally finished matches enter wins, draws and losses. Action-limit stops, cancellations, capacity limits and API failures remain separate: they are neither losses nor draws. Paired scores include a seat-swapped group only when both matches naturally finish. A failed endpoint stops the batch without retries because the match needs both models. Do not report a capped game as a draw or turn a connection failure into a model-strength score; small samples are not an Elo rating.

Every decision retains before/after state, action ID, legal candidates, state hashes, UTC timestamps and preparation, API, action, evidence and total times. Grouped decisions additionally retain each subrequest's level, group, candidates, exact request, validated choice, error and timing. Reports distinguish request latency from the sum for a decision and show mean/P50/P95. Optional PNGs are browser-rendered evidence, not operating-system screenshots or mouse-coordinate grounding.

All evaluation and exports run in this tab. Keys pass through the local extension to their associated model endpoint, never through a Chan Lab proxy, and are not saved to browser storage or reports. Arena responses retain only a validated action and usable numeric timing; raw service responses and authorization headers are discarded. Extension endpoint permissions and provider charges apply. See the request adapter, match runner and rules implementation.

2048 and Snake

Open game evaluation →. The browser-games-v1 protocol uses independent deterministic environments with common episode seeds. Short runs one episode with at most 32 actions; Medium runs three with at most 100 actions each; Long runs five with at most 200 each. APIs alternate, rotating the first API every step. No movement occurs while waiting for a model.

2048 retains the historical engine’s spawn and merge rules and stops at 2048 or no legal move. Snake uses a 10 × 10 grid, starts with three cells facing right, and scores one point per food. Reversal is forbidden; collision directions remain selectable. The tail vacates on moves without food, and fatal actions count as a step. There is no starvation rule.

Text and image inputs expose the present board and rules, never the seed, random generator state or future food. PNG evidence is a browser-rendered board, not operating-system capture or coordinate-click evidence. Each action logs before/after states, UTC timestamps and request, preparation, action, evidence and total durations. Reports compare completed episodes on common seeds. Failed, cancelled and unrun episodes stay separate; HTTP latency includes measured failed and cancelled requests.

A few game trajectories are diagnostic. Common seeds guarantee reproducible starts, not identical later boards after different decisions. Survival without food can inflate capped Snake survival, so read it alongside score. Keep the tab open and export before leaving; connection keys and results are not shared across tabs.

Texas Holdem · 2–10 model seats

We reuse @hivetech/poker-engine 1.0.1 under the MIT license. Each match is one independent hand with 200 chips per seat, blinds 1/2 and no rake. The legal menu offers fold/check/call and minimum, half-pot, pot and all-in bet sizes; it does not enumerate every integer wager. Player-safe inputs omit the seed, deck and other players’ hole cards and retain the last eight public actions. Full audit histories and private deals stay in local exports, never in model requests. Each complete N-seat rotation reuses a seeded deal. Net chips and net big blinds are the primary outcome; errors and stopped hands are unscored. These are simulated chips, not real-money bets.