BROWSER EVALUATION · 2026-09-22
Know what you are measuring.
Chan Lab evaluates multiple-choice answers, reproducible solo games and two-model chess matches. Your browser runs the environment, records and scores responses, and generates Markdown reports with Mermaid charts and measured data. Model inference runs at your selected endpoint. These tests do not measure mouse-coordinate grounding, independent web navigation or executing real-world tasks.
Your key and your data
- Keys live in page and local extension memory only. We do not write keys or evaluation results to cookies, localStorage, sessionStorage, IndexedDB, a database or an analytics service. Keys are cleared from the form when a run begins and released from the runner when it ends. Closing the page discards unsaved state; Other browser extensions and password managers are outside our control.
- There is no cloud evaluation backend or cloud API proxy. Chan Lab Connector uses a local extension service worker for model requests. Application scripts and datasets are served with the website. Cloudflare hosts these static files and may inject its Web Analytics script to collect website visits and performance metrics. Our application does not send API keys, model payloads or evaluation results to analytics. Hosting access logs and Cloudflare metrics are distinct from the evaluation records kept in your tab.
- Your selected provider receives your key, question payloads, IP address and ordinary request metadata. Its privacy policy and charges apply. Do not use an endpoint you do not trust. Redirects are refused, credentials are not put in URLs, browser cookies are omitted, and requests have no Referer header.
- For a multi-API comparison, each key is used only for its associated endpoint. Reports include configured endpoints and model names, but no Authorization headers. Returned content is scrubbed for that connection’s key before being retained.
- Exports are generated locally. They include questions, choices, answer keys, times, selected endpoint URLs, sanitized API responses and optionally rendered PNG cards. Sharing an export is your decision.
Dynamic question bank and replay
Questions are maintained in the public GitHub question bank. The browser loads its versioned JSON catalog on page load or when you click Refresh question bank, then checks the dataset schema and SHA-256. Compatible bank updates do not require a website release. A running evaluation keeps its loaded version. If the remote catalog is unavailable, the site clearly labels its bundled snapshot; hash or dataset validation failures block evaluation.
Each run draws a new sample without replacement, balanced across sets and topics. All APIs in the batch receive identical question IDs and option orders. A random seed, catalog and dataset hashes, source URLs and exact question IDs are exported in JSON and Markdown. Enter a seed under Advanced to reproduce a sample with the same bank version and settings. Short and Medium are nested only when their seeds match. Long is capped at 50 base questions so future growth cannot silently increase costs.
Question-bank downloads are public GET requests with no API key or cookies. Model keys go only to the selected model endpoint. The manifest and dataset contract is ready for another HTTP backend.
Read a real example
View the historical example report: verified full-run aggregates from 1,500 earlier attempts, plus 24 explicitly selected records. Download its Markdown/Mermaid source, JSON and CSV. It is an owner-provided public example from the earlier Mac/Playwright experiment, not a score from the current browser runner or a visitor’s uploaded result.
How to run a comparison
- Choose API format, Base URL, model, key and input mode. Accept your provider’s usage charges. For one API, run directly.
- For multiple APIs, click Add this API to comparison for each connection. Different models or modes may share a provider; keys remain associated with their own connection. Up to eight configurations per batch.
- Select a shared dataset, set and run length: Short uses 5 base questions, Medium 15, and Long up to 50 in a large bank. For a 10-question set or pilot, the sizes are 3 / 6 / 10. The default is one round. Advanced options can repeat the same questions with three fixed option orders, multiplying calls by three. The total request budget includes every API.
- Keep the tab open. Requests run sequentially; the first API rotates on each question. Each request has one attempt, with no retries and no previous answers fed back.
- Download Report .md for a complete Markdown report, data tables and Mermaid charts. JSON and CSV contain exact timings. Evidence ZIP adds the report and per-question PNGs. No extra model calls write the report.
Datasets and answer keys
The 10-question calculus pilot is separate from the 50-question expanded calculus bank. The expanded bank uses 10 topic templates, each with five parameterizations. The 50 decision scenarios cover parallel scheduling, prerequisites and resources, logical constraints, ordering, and plan changes. Prompts are English; answer explanations retain the original Chinese notes. We preserve the original question text, options and answers.
The two larger banks have three predetermined option permutations per question. These are 50 base questions, not 150 independent questions. Each JSON download contains the public answer key and source hashes. The evaluator hashes the dataset against the published manifest before running. Calculus answers were symbolically checked; scenario options were checked by enumeration and an independent state/schedule validator in the original experiment. This does not establish external difficulty calibration.
Answers and explanations never enter the model request. They remain available to anyone downloading the dataset, so this is a reproducible self-evaluation tool, not a cheating-resistant leaderboard. For scheduling questions, all listed schedules may be legal, while only one finishes earliest. An incorrect choice may simply be suboptimal.
Multiple-choice inputs and API adapters
- Text: task instructions, question ID/topic, question text and all four options.
- Image only: common task instructions and a PNG question card. Question text and options are in the image only.
- Text + image: both forms of the same question. This is not an equal-token comparison.
- Jev: the native
/systemoneChoice API; text only. - OpenJev:
/decidewith premise, four letter hypotheses and optional image_base64, using the local OpenJev wrapper’s response format. The server selects its model. - OpenAI-compatible:
/chat/completions, stream=false and max_tokens=128. It must return a single uppercase A/B/C/D in message.content. No response-format extension is required. Other wording, reasoning text, empty output and truncation are invalid answers. Temperature is left at the provider default. An API rejecting max_tokens needs its own compatible adapter; not every “OpenAI-compatible” endpoint accepts the same fields.
Image cards are browser-rendered canvases, 900 pixels wide with enough height for the complete question, using a 22-pixel body font. The exact ungraded PNG is sent to each image-enabled API. Result cards are drawn after scoring. They are not operating-system screenshots or evidence of a trusted mouse click. This rendering and the browser request environment differ from the original Mac/Playwright runs: do not merge the two experiments into one ranking.
Short, medium and long
Short and Medium use a deterministic, nested sample: first balance selected sets, then topics, with base ID as a stable tie-breaker. They do not simply take the first few questions. All APIs receive the same question IDs; the selected base questions repeat across option rounds. Reports record the preset, sampling version and complete question plan. A short check is not a full-dataset score.
Before measurements exist, the time estimate assumes 1–6 seconds per request and multiplies by the total calls across APIs. This is a planning range, not a speed guarantee. During a run, remaining time is estimated from measured per-API trial durations; subsequent plans can use matching measurements in this tab. There are no extra probe requests. Font and color theme preferences are held only in page memory.
Scoring and time definitions
- Main accuracy: correct / scored attempts. HTTP errors, CORS/network errors, timeouts and invalid responses count as failures.
- Valid-answer accuracy: correct / valid responses. It supplements, rather than replaces, the main metric.
- Partial runs: unattempted questions are not counted as wrong. User-cancelled requests are listed but excluded from scored accuracy and latency. Every report shows planned, attempted and scored counts. A partial batch does not declare a full-benchmark winner.
- API time: a monotonic browser timer from immediately before fetch through body reading, JSON parsing and choice validation. Includes network and service time. All scored failures remain in mean/P50/P95. Quantiles use linear interpolation; valid-only average is also shown.
- Other timing: UTC start/request/response/capture timestamps; card preparation, evidence rendering and full trial duration. UI table rendering and file downloads are outside the trial duration. OpenJev’s reported forward time, where present, is provider-reported and is not substituted for browser time.
- Comparisons: APIs receive the same question/option version. We show per-API totals, per-set/per-round tables, and a supplemental common-valid subset. Hardware, network, API adapters and provider defaults are uncontrolled. No significance test or general capability claim is made.
Local browser connector
Install Chan Lab Connector before running. Every run checks extension availability and endpoint permission before sending credentials. Jev rejects direct web-page CORS requests; the connector instead uses authorized extension background requests. Model inference stays at the selected API; all scoring and reports stay in this tab.
Jev permission is included; other HTTPS API origins require explicit permission in the extension. No automatic cloud fallback. Connection, authentication and timeout failures remain visible and stop affected requests. End-to-end API timings include extension messaging overhead.
Inspect it yourself
Download the static site source or inspect API adapters, browser runner, report generator and dataset hashes. In DevTools → Network, model calls should target only your chosen endpoint; no Key or model result is sent to this website. The ZIP library is vendored locally with its license.
Model arena: board games
Open the model arena →. All seats are models. Add two model-seat configurations; the same model may occupy both with independent settings. The seats alternate on one shared board; each request uses the active model seat's endpoint, key and input mode. A rules engine supplies all legal moves in a stable order, without a chess search engine, move ratings or candidate filtering. Waiting for an API changes neither the board nor a chess clock.
Short schedules 2 matches of at most 32 plies each; Medium 6 matches of 128 plies; Long 10 matches of 256 plies. One ply is one model seat's action. Every pair of matches swaps the model seats. All use the standard starting board: the seed identifies rotation groups and does not randomize openings. More games from the same start are not independent opening samples.
The rules version is chess-js-1.4-auto-draw-v1. Checkmate, stalemate, castling, en passant and all four promotions follow chess.js 1.4.0. Threefold repetition, the 50-move rule (100 half-moves without a pawn move or capture), and insufficient material detected by the library end automatically in a draw. This is the site's automatic-draw variant, not the complete FIDE claim procedure. There is no pass or resignation action.
Other supported rules
- Go · 9 × 9:
go9-area-playout-superko-v1. Black first, White +7.5; no suicide and positional superko. Two consecutive passes end area scoring on the board as it stands. Remaining stones are not automatically declared dead: capture before passing. This simplified play-out variant omits tournament dead-stone agreement and pass-stone rules. Coordinates a–i include i. See implementation and the AGA reference for the rules it differs from. - Xiangqi:
xiangqi-standard-threefold-v1. Red first; all seven piece types, palace/river constraints, horse legs, elephant eyes, cannon screens and facing generals are checked. Checkmate and stalemate both lose. The site automatically draws on three occurrences of the same board and side to move; it does not adjudicate perpetual checking/chasing or add a no-capture draw. Coordinates a0–i9. See implementation and WXF movement rules. - Chinese checkers:
checkers-2p-standard-hop-v1. Two opposite ten-piece camps on a 121-hole star. One adjacent step or a chain of adjacent-piece hops; no captures or long jumps. The other camps may be crossed or occupied. After entering its target camp, a piece cannot leave, including within a hop chain. Only actual occupation of all ten target holes wins; no anti-blocking swaps or early wins. Threefold repetition or no legal move is a site-rule draw. Equivalent routes share one endpoint action and a canonical shortest path. Axial q,r coordinates identify holes. See implementation and Hasbro instructions.
Rules and results are versioned by game. Do not pool outcomes across these different games or scoring variants. The rule engine never selects a move for strategic strength. Chess uses UCI move IDs; other games retain their own coordinate IDs and exact histories.
Action selection and request budgets
- All legal moves:
flat-legal-actions-v1sends the complete candidate list in one request. It stops before sending if that list exceeds the endpoint's configured capacity. It never truncates the list. - Groups of eight:
grouped-choice-v1splits the stable candidate list into groups of at most eight. The model chooses a winner from each group, then compares those winners through further levels until one remains. Single-candidate groups advance without a request. All requests see the same position and public history; earlier model replies and group explanations are not appended. This changes the decision protocol and can change choices; compare it separately from flat selection. - Per-decision limit: the site caps a decision at 64 subrequests and calculates the full requirement before making the first call. A capacity or request-budget failure stops that decision without dropping candidates. Default candidate capacities are Jev 255, OpenJev 8 and OpenAI-compatible 255. Jev's 255 limit cannot be raised. Raise another endpoint's configured capacity only if its server supports it.
- Forced action: when there is exactly one legal move, the browser applies and records it with zero model requests in either protocol. This is not a measured model choice. A grouped decision over 20 candidates normally needs 4 requests; flat selection needs 1.
Model inputs, results and evidence
Text sends the assigned seat, full current public observation, rules and legal candidate IDs with neutral move descriptions. Text + image adds the same position's PNG. Image-only text contains rules, candidate IDs and labels, and minimal game/seat/turn identifiers; it contains no FEN, board layout or move history that could reconstruct the board. The PNG includes the complete board, special-move rights, counters and full move history, with height extended instead of truncating history. Jev is text-only. Image and text payloads are different representations, not an equal-token control.
Only naturally finished matches enter wins, draws and losses. Action-limit stops, cancellations, capacity limits and API failures remain separate: they are neither losses nor draws. Paired scores include a seat-swapped group only when both matches naturally finish. A failed endpoint stops the batch without retries because the match needs both models. Do not report a capped game as a draw or turn a connection failure into a model-strength score; small samples are not an Elo rating.
Every decision retains before/after state, action ID, legal candidates, state hashes, UTC timestamps and preparation, API, action, evidence and total times. Grouped decisions additionally retain each subrequest's level, group, candidates, exact request, validated choice, error and timing. Reports distinguish request latency from the sum for a decision and show mean/P50/P95. Optional PNGs are browser-rendered evidence, not operating-system screenshots or mouse-coordinate grounding.
All evaluation and exports run in this tab. Keys pass through the local extension to their associated model endpoint, never through a Chan Lab proxy, and are not saved to browser storage or reports. Arena responses retain only a validated action and usable numeric timing; raw service responses and authorization headers are discarded. Extension endpoint permissions and provider charges apply. See the request adapter, match runner and rules implementation.
2048 and Snake
Open game evaluation →. The browser-games-v1 protocol uses independent deterministic environments with common episode seeds. Short runs one episode with at most 32 actions; Medium runs three with at most 100 actions each; Long runs five with at most 200 each. APIs alternate, rotating the first API every step. No movement occurs while waiting for a model.
2048 retains the historical engine’s spawn and merge rules and stops at 2048 or no legal move. Snake uses a 10 × 10 grid, starts with three cells facing right, and scores one point per food. Reversal is forbidden; collision directions remain selectable. The tail vacates on moves without food, and fatal actions count as a step. There is no starvation rule.
Text and image inputs expose the present board and rules, never the seed, random generator state or future food. PNG evidence is a browser-rendered board, not operating-system capture or coordinate-click evidence. Each action logs before/after states, UTC timestamps and request, preparation, action, evidence and total durations. Reports compare completed episodes on common seeds. Failed, cancelled and unrun episodes stay separate; HTTP latency includes measured failed and cancelled requests.
A few game trajectories are diagnostic. Common seeds guarantee reproducible starts, not identical later boards after different decisions. Survival without food can inflate capped Snake survival, so read it alongside score. Keep the tab open and export before leaving; connection keys and results are not shared across tabs.
Texas Holdem · 2–10 model seats
We reuse @hivetech/poker-engine 1.0.1 under the MIT license. Each match is one independent hand with 200 chips per seat, blinds 1/2 and no rake. The legal menu offers fold/check/call and minimum, half-pot, pot and all-in bet sizes; it does not enumerate every integer wager. Player-safe inputs omit the seed, deck and other players’ hole cards and retain the last eight public actions. Full audit histories and private deals stay in local exports, never in model requests. Each complete N-seat rotation reuses a seeded deal. Net chips and net big blinds are the primary outcome; errors and stopped hands are unscored. These are simulated chips, not real-money bets.