I built a quality gate for AI deliveries. Here's how I specified it

I have 69 AI-delivered items for my Health tracking app in my own history and, until this month, no honest way to say which ones were good. I had an impression. Impressions are what you get when you review AI output at the end of a long day.

I've been researching quality gates and how I can trust and evaluate AI deliverables based on my own data. So, I looked at the question I actually have when an agent finishes a task. I need to know whether this delivery did what the ticket asked, whether it finished the job, and whether I can trust the PR description

What I thought I needed

I started with a skill called create-benchmark. The idea was simple: run the same slice of my daily work through different models, score them, and pick the best one. A leaderboard for my own use.

That lasted about five phases until I realized what I actually was looking for. Then I changed the brief: I don't need to test models and their scores. I need a benchmark that works as a quality gate for what AI delivers in my projects.

A leaderboard asks which model is better on average. A gate asks whether this delivery, today, should be merged. And that's the difference between them. And the question the gate asks is the one I face every time an agent opens a PR.

What the gate asks

Four questions, taken from how I actually review.

Did the AI deliver what the ticket asked? Did it go all the way, or stop at the first green checkmark? Do the results match the ticket? Do the PR title and description match the diff?

Each question became a weighted dimension. So I borrowed what the industry already measures, such as resolved-issue rates, requirement traceability, coverage on new code, and small-change practice, instead of inventing my own.

Today they are: requirement coverage, completion, verification, security, scope, PR fidelity, conventions, and general practices. The score is a weighted mean over the dimensions that apply. 80 or more is a PASS, 65 to 79 is a WARN, below 65 is a FAIL. Any single dimension under 40 fails the delivery on its own, so a strong dimension can't hide a broken one.

On top of that, there are hard blockers that fail regardless of score: an acceptance criterion marked missing, a failing test or typecheck, a leftover .only, a secret in the added lines, and the AI claiming tests passed when the transcript shows they didn't run.

Deterministic first, judge second

This is the decision I'd defend hardest. Every run starts with the deterministic stage: test and typecheck commands, scope, conventions, completion scan, PR rules, secrets. If anything there fails or raises a blocker, the LLM judge never runs.

Two reasons. A judge costs money, and scoring a delivery that already fails its own tests is wasted money. And a judge's opinion on broken work is noise that makes the report harder to read. When the gate stops early, the judged dimensions show as skipped, and the verdict is FAIL with zero judge calls.

Using it

I initially made it a CLI command, but since I'm not much of a cli type-of-dev myself, I made a UI for it and I can start a run from there. That's good, but not good enough. It wouldn't act upon the triggering problem: an agent opening a PR. And what matters to me is running it right before that PR is created.

Then I created a hook.

The hook runs before gh pr create, because that's when I want to catch a misleading PR or an unfinished delivery.

Dogfooding

I ran the gate on its own repo. It found real things: functions over 20 lines, new exports with no test touching them, magic numbers, an empty catch block.

It also found false positives in its own analyzers. The ReDoS detector flagged two regexes in the gate's own source as nested quantifiers and blocked the run with two high findings. The detector was right to be suspicious of that shape and wrong about those patterns. I fixed the detector.

Security came later at weight 20, covering IDOR, SQL injection, ReDoS, XSS, weak crypto, and vulnerable dependencies, with critical and high findings blocking on their own. General practices came in at 5 and are exempt from the floor: they should nudge a score, never fail a delivery alone.

Summarizing

Now, I needed a more complete UI, so I made a dashboard to list projects and active runs. Opening one shows the delivery moving through tests, deterministic checks, the gate, the LLM judge, and the scorecard. The current check and its evidence appear as the run progresses.

The UI showed me all I wanted: the exact check, evidence, and rule behind a verdict, then let me correct either the delivery or the gate. Sometimes the tool finds a real problem. Sometimes its own rule is wrong. I need to be able to tell which.

This very "simple" tool now helps me check whether my AI deliverables are good, safe, and complete enough.

Measuring made me go from empiric evaluation to actual trustful metrics (though they might need calibration)