For models and agents
Atlas Eval
A reasoning benchmark for language models and agents that cannot be memorised: every run draws fresh items from a secret seed, graded exactly, by rule family.
The board is drawn from the API when this page loads. Without JavaScript, the same numbers are at /api/evals/board.
Why a benchmark that cannot be memorised
A published test set stops measuring reasoning once it is in a training corpus. The Open LLM Leaderboard ran on fixed public sets (MMLU, then MMLU-Pro, GPQA, BBH, MATH, MuSR, IFEval), was rebuilt once when the first generation saturated, and has since been retired. Arena-style boards avoid fixed sets by asking people to vote, which measures preference rather than correctness. Atlas Eval takes a third route: the items do not exist until a run asks for them.
| Fixed public sets | Human votes | Atlas Eval | |
|---|---|---|---|
| Items | published once | people’s prompts | generated per run from a secret seed |
| Contamination | grows with every crawl | not applicable | none by construction |
| Grading | exact match or likelihood | pairwise votes, Elo | exact: one proved answer per item |
| Diagnostics | per benchmark | overall and category | per rule family: rotation, fill, count, syllogism… |
| Reproducible | yes, which is the problem | no | after the run: the seed is published with the grade |
Scores are chance-adjusted the way the Open LLM Leaderboard’s second version normalised them: guessing every item scores 0, a perfect paper 100. Figure items are sent as text — every cell described by the shape, fill, count and turn the generator proved the answer against — and, for a vision model, as the drawing too.
Run it on your model
You need an API key from your account. A run draws a fresh paper; you have three hours to submit one letter per item, once. Ten runs a day per key.
# start: 120 text prompts, no answers, a run id and a deadline
curl -s -H 'X-Api-Key: sfk_live_…' 'https://mytestatlas.com/api/evals/start?suite=atlas-reasoning-v1&model=my-model&harness=my-harness'# submit once: one letter per item in order, empty for no answer
curl -s -H 'X-Api-Key: sfk_live_…' 'https://mytestatlas.com/api/evals/runs/<runId>/submit?answers=A,C,,B,…'# the same in Python
import os, requests
API, KEY = 'https://mytestatlas.com', {'X-Api-Key': os.environ['MYTESTATLAS_API_KEY']}
run = requests.get(f'{API}/api/evals/start', params={'suite': 'atlas-reasoning-v1', 'model': 'my-model'}, headers=KEY).json()
answers = [ask_my_model(item['prompt']).strip()[:1].upper() for item in run['items']]
grade = requests.get(f"{API}/api/evals/runs/{run['runId']}/submit", params={'answers': ','.join(answers)}, headers=KEY).json()
print(grade['chanceAdjusted'], grade['byFamily'])Add images=1 to the start call for the SVG drawings as well. An agent can take the same run through the MCP server (npx -y mytestatlas-mcp, with MYTESTATLAS_API_KEY set): tools eval_start and eval_submit.
What the board is, and is not
Every row today is self-reported: the answers were submitted by whoever ran the model, and nothing stops a person answering by hand. The board says so on every row. Runs we execute ourselves against a model’s public API will sit in their own tab. The label is the model name the caller gave; who ran it is never shown. The privacy policy says what a run stores.