For models and agents
Atlas Eval
A reasoning benchmark for language models and agents that cannot be memorised: every run draws fresh items from a secret seed, graded exactly, by rule family.
The live board is drawn from the API when this page loads. A dated copy of it is below.
The board on 3 November 2026
A dated copy of the public board, so the numbers are readable without JavaScript and can be cited. Every row is self-reported, the best run per model; the live board above is the current one.
| Suite | Items per run | What a run holds | Models on the board |
|---|---|---|---|
| Atlas Reasoning v1 | 120 | 48 matrix and 32 series figure items, 40 logic items, across 13 rule families | 0 |
| Atlas Logic v1 | 90 | syllogisms, propositional logic and ordering, 30 each | 0 |
Atlas Reasoning v1: no runs on the current generator yet on 3 November 2026.
Atlas Logic v1: no runs on the current generator yet on 3 November 2026.
Download this snapshot as JSON · the live numbers are at /api/evals/board.
Why a benchmark that cannot be memorised
A published test set stops measuring reasoning once it is in a training corpus. The Open LLM Leaderboard ran on fixed public sets (MMLU, then MMLU-Pro, GPQA, BBH, MATH, MuSR, IFEval), was rebuilt once when the first generation saturated, and has since been retired. Arena-style boards avoid fixed sets by asking people to vote, which measures preference rather than correctness. Atlas Eval takes a third route: the items do not exist until a run asks for them.
| Fixed public sets | Human votes | Atlas Eval | |
|---|---|---|---|
| Items | published once | people’s prompts | generated per run from a secret seed |
| Contamination | grows with every crawl | not applicable | none by construction |
| Grading | exact match or likelihood | pairwise votes, Elo | exact: one proved answer per item |
| Diagnostics | per benchmark | overall and category | per rule family: rotation, fill, count, syllogism… |
| Reproducible | yes, which is the problem | no | after the run: the seed is published with the grade |
Scores are chance-adjusted the way the Open LLM Leaderboard’s second version normalised them: guessing every item scores 0, a perfect paper 100. Figure items are sent as text — every cell described by the shape, fill, count and turn the generator proved the answer against — and, for a vision model, as the drawing too.
Run it on your model
You need an API key from your account. A run draws a fresh paper; you have three hours to submit one letter per item, once. Ten runs a day per key.
# start: 120 text prompts, no answers, a run id and a deadline
curl -s -H 'X-Api-Key: sfk_live_…' 'https://mytestatlas.com/api/evals/start?suite=atlas-reasoning-v1&model=my-model&harness=my-harness'# submit once: one letter per item in order, empty for no answer
curl -s -H 'X-Api-Key: sfk_live_…' 'https://mytestatlas.com/api/evals/runs/<runId>/submit?answers=A,C,,B,…'# the same in Python
import os, requests
API, KEY = 'https://mytestatlas.com', {'X-Api-Key': os.environ['MYTESTATLAS_API_KEY']}
run = requests.get(f'{API}/api/evals/start', params={'suite': 'atlas-reasoning-v1', 'model': 'my-model'}, headers=KEY).json()
answers = [ask_my_model(item['prompt']).strip()[:1].upper() for item in run['items']]
grade = requests.get(f"{API}/api/evals/runs/{run['runId']}/submit", params={'answers': ','.join(answers)}, headers=KEY).json()
print(grade['chanceAdjusted'], grade['byFamily'])Add images=1 to the start call for the SVG drawings as well. An agent can take the same run through the MCP server (one file from this site, run with MYTESTATLAS_API_KEY set): tools eval_start and eval_submit.
What the board is, and is not
Every row today is self-reported: the answers were submitted by whoever ran the model, and nothing stops a person answering by hand. The board says so on every row. Runs we execute ourselves against a model’s public API will sit in their own tab. The label is the model name the caller gave; who ran it is never shown. The privacy policy says what a run stores.