BROWSER-CHOICE-V1 · 2026-09-21

Know what you are measuring.

Chan Lab evaluates a model’s choice among four explicit answers. It records the response, scores it locally, and produces a deterministic Markdown report with Mermaid charts and the measured data. It does not test mouse-coordinate grounding, independent web navigation or executing real-world tasks.

Your key and your data

• Keys live in page memory only. We do not write keys or evaluation results to cookies, localStorage, sessionStorage, IndexedDB, a database or an analytics service. Keys are cleared from the form when a run begins and released from the runner when it ends. Closing the page discards unsaved state; browser extensions and password managers are outside our control.
• There is no evaluation backend, API proxy, service worker or advertising script. Application scripts and datasets are served with the website. Cloudflare hosts these static files and may inject its Web Analytics script to collect website visits and performance metrics. Our application does not send API keys, model payloads or evaluation results to analytics. Hosting access logs and Cloudflare metrics are distinct from the evaluation records kept in your tab.
• Your selected provider receives your key, question payloads, IP address and ordinary request metadata. Its privacy policy and charges apply. Do not use an endpoint you do not trust. Redirects are refused, credentials are not put in URLs, browser cookies are omitted, and requests have no Referer header.
• For a multi-API comparison, each key is used only for its associated endpoint. Reports include configured endpoints and model names, but no Authorization headers. Returned content is scrubbed for that connection’s key before being retained.
• Exports are generated locally. They include questions, choices, answer keys, times, selected endpoint URLs, sanitized API responses and optionally rendered PNG cards. Sharing an export is your decision.

Read a real example

View the historical example report: verified full-run aggregates from 1,200 earlier attempts, plus 20 explicitly selected records. Download its Markdown/Mermaid source, JSON and CSV. It is an owner-provided public example from the earlier Mac/Playwright experiment, not a score from the current browser runner or a visitor’s uploaded result.

How to run a comparison

1. Choose API format, Base URL, model, key and input mode. Accept your provider’s usage charges. For one API, run directly.
2. For multiple APIs, click Add this API to comparison for each connection. Different models or modes may share a provider; keys remain associated with their own connection. Up to eight configurations per batch.
3. Select a shared dataset, set and run length: Short uses 5 base questions, Medium 15, and Long all 50 in a large bank. For a 10-question set or pilot, the sizes are 3 / 6 / 10. The default is one round. Advanced options can repeat the same questions with three fixed option orders, multiplying calls by three. The total request budget includes every API.
4. Keep the tab open. Requests run sequentially; the first API rotates on each question. Each request has one attempt, with no retries and no previous answers fed back.
5. Download Report .md for a complete Markdown report, data tables and Mermaid charts. JSON and CSV contain exact timings. Evidence ZIP adds the report and per-question PNGs. No extra model calls write the report.

Datasets and answer keys

The 10-question calculus pilot is separate from the 50-question expanded calculus bank. The expanded bank uses 10 topic templates, each with five parameterizations. The 50 decision scenarios cover parallel scheduling, prerequisites and resources, logical constraints, ordering, and plan changes. Prompts are English; answer explanations retain the original Chinese notes. We preserve the original question text, options and answers.

The two larger banks have three predetermined option permutations per question. These are 50 base questions, not 150 independent questions. Each JSON download contains the public answer key and source hashes. The evaluator hashes the dataset against the published manifest before running. Calculus answers were symbolically checked; scenario options were checked by enumeration and an independent state/schedule validator in the original experiment. This does not establish external difficulty calibration.

Answers and explanations never enter the model request. They remain available to anyone downloading the dataset, so this is a reproducible self-evaluation tool, not a cheating-resistant leaderboard. For scheduling questions, all listed schedules may be legal, while only one finishes earliest. An incorrect choice may simply be suboptimal.

Inputs and API adapters

• Text: task instructions, question ID/topic, question text and all four options.
• Image only: common task instructions and a PNG question card. Question text and options are in the image only.
• Text + image: both forms of the same question. This is not an equal-token comparison.
• Jev: the native /systemone Choice API; text only.
• OpenJev: /decide with premise, four letter hypotheses and optional image_base64, using the local OpenJev wrapper’s response format. The server selects its model.
• OpenAI-compatible: /chat/completions, stream=false and max_tokens=128. It must return a single uppercase A/B/C/D in message.content. No response-format extension is required. Other wording, reasoning text, empty output and truncation are invalid answers. Temperature is left at the provider default. An API rejecting max_tokens needs its own compatible adapter; not every “OpenAI-compatible” endpoint accepts the same fields.

Image cards are browser-rendered canvases, 900 pixels wide with enough height for the complete question, using a 22-pixel body font. The exact ungraded PNG is sent to each image-enabled API. Result cards are drawn after scoring. They are not operating-system screenshots or evidence of a trusted mouse click. This rendering and the browser request environment differ from the original Mac/Playwright runs: do not merge the two experiments into one ranking.

Short, medium and long

Short and Medium use a deterministic, nested sample: first balance selected sets, then topics, with base ID as a stable tie-breaker. They do not simply take the first few questions. All APIs receive the same question IDs; the selected base questions repeat across option rounds. Reports record the preset, sampling version and complete question plan. A short check is not a full-dataset score.

Before measurements exist, the time estimate assumes 1–6 seconds per request and multiplies by the total calls across APIs. This is a planning range, not a speed guarantee. During a run, remaining time is estimated from measured per-API trial durations; subsequent plans can use matching measurements in this tab. There are no extra probe requests. Font and color theme preferences are held only in page memory.

Scoring and time definitions

• Main accuracy: correct / scored attempts. HTTP errors, CORS/network errors, timeouts and invalid responses count as failures.
• Valid-answer accuracy: correct / valid responses. It supplements, rather than replaces, the main metric.
• Partial runs: unattempted questions are not counted as wrong. User-cancelled requests are listed but excluded from scored accuracy and latency. Every report shows planned, attempted and scored counts. A partial batch does not declare a full-benchmark winner.
• API time: a monotonic browser timer from immediately before fetch through body reading, JSON parsing and choice validation. Includes network and service time. All scored failures remain in mean/P50/P95. Quantiles use linear interpolation; valid-only average is also shown.
• Other timing: UTC start/request/response/capture timestamps; card preparation, evidence rendering and full trial duration. UI table rendering and file downloads are outside the trial duration. OpenJev’s reported forward time, where present, is provider-reported and is not substituted for browser time.
• Comparisons: APIs receive the same question/option version. We show per-API totals, per-set/per-round tables, and a supplemental common-valid subset. Hardware, network, API adapters and provider defaults are uncontrolled. No significance test or general capability claim is made.

HTTPS, CORS and local models

The public HTTPS site requires an HTTPS endpoint that allows cross-origin requests from this site, including POST and the Content-Type and Authorization headers. A CORS/network failure stops that API’s remaining requests; other APIs can continue. Authentication errors (401/403), rate limits (429) and timeouts also stop that API to avoid repeating an invalid setup. No key is forwarded through our server to bypass these restrictions.

To use an HTTP localhost OpenJev server, download this site, unzip it, run python3 -m http.server 8788 --bind 127.0.0.1 from the extracted site directory, and open http://127.0.0.1:8788. Configure the local model server to allow that exact browser origin. The local copy permits loopback HTTP API addresses; it still requires CORS for a different port.

Inspect it yourself

Download the static site source or inspect API adapters, browser runner, report generator and dataset hashes. In DevTools → Network, model calls should target only your chosen endpoint; no Key or model result is sent to this website. The ZIP library is vendored locally with its license.