Sonnet beat Opus on my security evals for under half the cost

Sonnet 5.5 passed 30 of 30 attempts on my medium security tasks. Opus 5.5 passed 27, and cost $0.077 per task against $0.034.

Both numbers come from the leaderboard I added to the quality gate this week. It ranks models on tasks I wrote from my own projects and prices every attempt, so "which one should I use" stops being a feeling.

Why a gate grows a leaderboard

The quality gate scores what an AI agent delivered against its ticket. After a few weeks of using it I kept asking a different question: if I swap the model behind the agent, does the score move, and what does it cost me? Public benchmarks cannot answer that, because they do not use my repos, my conventions or my habits.

So the gate got a Model evals area. It runs a set of agent setups against a set of tasks, several times each, and ranks them.

A contender is more than a model

What gets ranked is a contender: a harness (Claude Code or Codex), a model, and its settings. Two contenders that differ only in the model compare models. Two that differ only in the harness compare tools.

The settings are the part I use most. A contender can carry its own system prompt, turn skills on, and switch on "use my setup", which hands it my global instructions, hooks and skills. All of it is off by default. The agent runs in a clean config directory with only the credentials copied in, because a comparison where one side carries my personal setup is not a comparison. I wanted the setup itself to be testable, so it is one more contender: "Sonnet 5.5, skills and my setup on" sits in the table next to plain Sonnet.

The tier is the other setting. Codex has a priority tier that bills at double the standard rate, and a contender can be pinned to it.

Money comes from a price table in US$ per 1M tokens, with input, output, cache read and cache write for each model, plus the priority rates where the model has them. Opus 5.5 is $4 in and $20 out, Haiku 5.5 is $0.10 and $0.50. Every attempt records the tokens it used and the rates it was billed at, so the cost is the API-equivalent price even when the agent ran on a subscription. On a subscription the real cost is zero, and a ranking by cost would be useless.

Two kinds of task

There are 7 datasets and 56 tasks today, in two kinds.

A code task is a repo, a commit to start from and a ticket. The agent works in a clone with every remote removed, and the gate scores its delivery the way it scores any other delivery. These come from real commits of mine that held up.

A prompt task has no repo and no tools. It is a question with an answer key, and the key comes in three forms: a rubric that a fixed judge (Haiku 5.5, pass at 70) applies, an exact or regex answer, or a script that runs the answer's code. Each task has a time limit and a dollar limit. An attempt that hits either is marked timeout or over budget, never as a quality failure, because a stopped attempt says nothing about how good the model is.

Every task runs k times, 3 by default. One lucky answer should not rank anyone. The leaderboard sorts by pass rate, then mean score, and shows consistency next to them: how much the score of a task moves between its repetitions.

A confession: the code datasets have barely run. The only attempt on record is one Haiku run on a backend task, 94.8 at $0.023 and 2m 16s. Code attempts install, build and test a whole checkout, and I have not run a full code dataset yet. Everything below is prompt tasks.

Security tasks with a trap in each

I wrote three security datasets, easy with 10 tasks, medium with 10 and hard with 8, all on synthetic Node and Supabase code. Most are scored by scripts that run the answer against attack payloads rather than by a judge, so a model cannot talk its way to a pass.

Medium is where each task plants a safe decoy next to the real hole: an ID check that looks missing but is covered by a trusted filter, injection in LIKE, ORDER BY and LIMIT, a zip that escapes through a symlink, an SSRF through a redirect. Hard goes after second-order SQL injection, a cache key that drops the tenant, a balance race that has to be tested with two module instances, and a prompt injection split across a diff.

Each check had to fail the vulnerable code and the plausible half-fixes, and pass a reference fix, before I trusted it. The proof runs passed 42 of 42 on medium and 27 of 27 on hard.

The rubric that was wrong

The easy dataset has a Supabase row-level-security task. My first rubric listed four flaws, and one of them was an UPDATE policy with a USING clause and no WITH CHECK, on the theory that an owner could rewrite the user id and hand the row to someone else.

It is wrong. When an UPDATE policy has no WITH CHECK, Postgres applies the USING expression to the new row as well. The policy is safe. Answers that pushed back on my claim were being penalised.

The rubric now says the policy is safe, gives full marks for adding an explicit WITH CHECK anyway, and takes 10 points off an answer that claims the hole exists. I did not re-measure the old runs. A dataset's version is a hash of its content, so a fix makes a new version and rankings never mix runs from before and after it.

What the first numbers say

On medium, Sonnet 5.5 passed 100% at $0.034 per task and 19s, Opus 5.5 90% at $0.077 and 25s, and Codex with gpt-6-luna 70% at $0.003 and 8.4s. The Opus spread was ±0.0: every task scored the same on all three repetitions, so it missed one task three times out of three.

On hard, Sonnet and Opus tie at a 92% pass rate, with Sonnet ahead on mean score (95.4 against 93.1) and steadier (±1.9 against ±7.3). The row-level-security task is where everyone drops: in the larger run, each of the three plain Claude contenders passed it once in three tries.

The setups did not earn their keep. Sonnet with skills and my setup passed 87% on 15 attempts and cost $0.051 per task, against 92% and $0.035 for plain Sonnet. Be careful with that comparison: the setup contenders only ran 5 of the 8 hard tasks, and k is 3. I am treating it as a reason to look again, not as a result.

Luna is the cheap end of the chart. The leaderboard shows output tokens instead of steps for prompt datasets, since there are no steps without tools, and Luna writes 583 tokens where Haiku writes 4.6k.

How many attempts at once

The run form has a concurrency field, 1 to 8, and a default that depends on the dataset: 2 for code, 4 for prompts. A code attempt runs the agent and then the gate's install, build and tests in its own checkout, which is heavy on the CPU. A prompt attempt only waits for the API. Pushing code attempts past 2 or 3 slows every attempt and skews the time column in the ranking, so the form says so next to the field.

Smaller things

• Two runs of the same delivery can now be compared: verdict and score change, each dimension, and the findings that appeared, disappeared or stayed. Findings are matched by check, rule and file, never by line, so moving code does not look like a new problem.
• A failed run can be re-run on the same branch and compared with the earlier one when it finishes.
• A failing run gets a fix prompt: what the delivery still needs for a pass, ready to paste into the agent.
• A local MCP server lets an agent ask the gate for a score while it is still working. Those runs are listed but kept out of baselines.
• Saving a new contender from the dashboard answered 400 until I fixed it. The server now makes the id.

What comes next

The next big run is a code dataset, with the setups that lost on prompts. If you rank models on your own work: how do you notice that the answer key is the thing that is wrong?