Every AI proposal has a slide that says a human reviews everything, and almost none of them can answer who that human is, what they see on screen, or what happens when they say no. If those 3 questions have no answers, the loop is a diagram. A working loop is a staffed role with a queue, a review surface, an escalation path, and a feedback wire into the eval suite.
The failure is easy to picture because it keeps happening. A returns-triage agent goes live in September with a review queue in front of every refund, and by week 6 the queue holds 340 items at 9am, the reviewer is an ops coordinator who also owns carrier escalations, and the approval rate is 99.7%.
Nobody lied. The team did put a human in the loop. They just never designed the job, so the job degraded into clicking approve at 8 seconds per item, and the control everyone pointed to in the risk review stopped controlling anything.
This post is a sequel to the evals piece, which argued that the golden set and the grader come before the agent. The reviewer role is the runtime half of the same system: evals tell you whether the model regressed, reviewers tell you whether it is wrong right now, and each one feeds the other.
Code review already solved this shape of problem
Engineering has run a human-in-the-loop system at scale for 2 decades. It is called code review, and the reason it works is that nobody treats it as a checkbox.
Google's engineering practices documentation specifies the role in writing: what a reviewer looks for, in what order, what the standard for approval is, and how fast a review should turn around. The reviewer is not whoever is free. The standard is not vibes.
Copy the structure, not the ceremony. An AI review workflow needs the same 4 artifacts a code review culture has.
- A written standard for what approve means, specific enough that 2 reviewers given the same item usually agree.
- A defined reviewer pool with domain ownership, not a rotating pool of whoever has slack that day.
- A turnaround expectation, because a blocking queue with no SLA becomes the pipeline's slowest stage silently.
- A record of every verdict with a reason, because unexplained rejections cannot tune anything downstream.
Teams that would never merge unreviewed code to a payments service will happily let an unreviewed model email 40,000 customers. The gap is not caution. The gap is that code review is a designed practice and AI review usually is not.
Who reviews: ownership beats availability
The reviewer must be the person accountable for the decision the model is making, or someone who reports to them. For returns triage that is an ops lead who owns refund policy. For generated product copy it is a merchandiser who owns the category.
Engineers make bad default reviewers here, for the same reason the policy team makes a bad reviewer of database migrations. An engineer can tell that the output parsed. Only the domain owner can tell that the output is wrong in a way that costs money or trust.
This has an uncomfortable staffing consequence: the loop consumes hours from your most knowledgeable operators, not your cheapest. Budget it like that from the start. A review workflow priced at intern rates gets staffed at intern judgment, and the whole point of the gate evaporates.
The reviewer is the person who owns the outcome, seeing the evidence, with the authority to say no and a wire that makes their no change the system. Remove any clause of that sentence and you have a checkbox again.
What the reviewer sees: the review surface is a product
Reviewer throughput and reviewer accuracy are both functions of the screen you give them. A queue that shows the raw customer thread plus the model's answer forces a full re-read of every case, which caps a careful reviewer at maybe 40 items a day and invites skimming past that.
A good review surface shows 5 things: the model's decision, the specific evidence it relied on, its confidence, the policy or rule it applied, and a diff against what will happen if approved. Evidence spans matter most, because they turn review from re-derivation into verification, and verification is 10x faster for the same accuracy.
Purpose-built tooling for this exists and is worth studying even if you build your own. LangSmith's annotation queues model the mechanics well: items enter a queue with instructions and rubric attached, reviewers reserve an item so 2 people never grade the same run, and reservations expire so an abandoned item returns to the pool.
On the open-source side, Label Studio gives you configurable review interfaces, per-annotator agreement tracking, and an API for pushing verdicts back out. The build-vs-buy call matters less than the requirement: verdicts must land somewhere structured, because a spreadsheet of reviewer opinions feeds nothing.
One rule regardless of tooling: the reviewer records a reason code with every rejection, from a closed enum you maintain. Free-text rejection notes feel richer and aggregate into nothing. Enum reason codes become the Pareto chart that tells you which failure mode to fix next.
Gating and sampling are different instruments
Most arguments about human review dissolve once you separate the 2 loops that get conflated under one name. A gate is a blocking review before the action executes. A sample is an audit of a percentage of actions after they executed.
| Property | Gate (before the action) | Sample (after the action) |
|---|---|---|
| Coverage | 100% of gated actions | A chosen fraction, weighted by risk |
| Latency cost | Every action waits on a human | Zero added latency |
| What it prevents | The bad action itself | Nothing; it detects and bounds |
| Failure mode | Queue backlog, rubber-stamping under load | A bad pattern runs until the sample catches it |
| Right for | Irreversible, high-value, precedent-setting actions | Reversible, high-volume, low-blast-radius actions |
| Scales by | Adding reviewers or narrowing the gate | Adjusting the sampling rate per slice |
The routing rule between them is the same one the returns and refunds world uses for automation generally. Money movement, permanent record changes, and anything precedent-setting go through the gate. Reversible and cheap-to-undo actions ship immediately and get sampled.
Sampling rates should be per-slice, not global. New product category, or a locale you launched last month: sample heavily. The workflow that has run 9 months with stable agreement: sample lightly and spend the reviewer hours where the uncertainty is.
The 2 loops: a blocking gate for irreversible actions, a sampled audit for reversible ones, both wiring verdicts back into the eval suite.
When the loop tightens, and when it loosens
Static review policies rot in both directions. Too tight and the queue backlog trains reviewers to rubber-stamp. Too loose and the sample misses a failure pattern for a month.
The right shape is a set of named triggers, agreed before launch, that move the dials automatically or at least loudly.
- Model version change, yours or the vendor's: gate widens to 100% on affected workflows until the eval suite and a reviewed batch both clear.
- Policy change, like a new return window or promo rule: affected slices go back behind the gate, because the model's training signal is now stale by definition.
- New slice in production, a category, locale or channel the golden set barely covers: gated until the slice has its own eval floor.
- Reviewer disagreement rate rising on a slice: sampling rate doubles there before anyone debates why.
- Sustained agreement above the threshold you set, measured per-slice over weeks: the gate narrows one notch, deliberately and with a record of who decided.
Loosening deserves as much ceremony as tightening. The evals post made the point that a gate approving roughly everything for a sustained period is either theater or a signal that the automation has earned a wider bound. The tightening triggers above are what make widening safe: you can loosen confidently because you know exactly what snaps the gate shut again.
The verdict wire: reviews must change the system
A review loop that only blocks bad actions is running at a third of its value. Every verdict is a labeled example produced by your most qualified labeler, on real production traffic, at the exact frontier where the model is uncertain.
Wire it accordingly. Rejected items go into the golden set with the corrected output as reference. Disagreement clusters become new eval slices. Reason-code frequencies drive the next prompt or policy revision, ranked by cost rather than by count.
This is the loop half that the golden set cannot do alone. Evals catch regressions against yesterday's known cases. Reviewers catch tomorrow's novel ones, and the wire between them is what turns a one-time dataset into a living one.
Google's People + AI Guidebook is useful on the design side of this wire: it treats feedback collection as something you design deliberately, deciding what signal you need, what the user gives you, and what the system does with it. The same discipline applies when the user giving feedback is your own reviewer.
Reviewer capacity is a queueing problem
Review workflows fail on arithmetic before they fail on judgment. If the agent emits 2,000 decisions a day, the gate covers 15% of them, and honest review takes 90 seconds per item, that is 7.5 reviewer-hours every day, before breaks, escalations, or the BFCM volume triple.
Do that arithmetic before launch, for peak volume rather than average. Then decide which lever moves when the queue exceeds capacity, because one of them will move whether you choose it or not.
- Narrow the gate: route a smaller, riskier subset to blocking review and push the rest to sampling.
- Improve the surface: cut seconds per item with better evidence display, which is an engineering task with measurable payoff.
- Add reviewers: the honest option, with the real cost that makes stakeholders reconsider scope.
- Let the queue age: the default option, and the one that quietly converts your gate into a latency tax with no control attached.
The fourth lever is what happened to the ops coordinator with 340 items at 9am. Nobody chose it. It was chosen by the absence of a capacity plan.
Review capacity gets decided at design time whether or not anyone decides it. When the queue can receive 2,000 items a day and the roster can honestly review 400, the other 1,600 are being approved by fatigue.
Guarding the guard: overreliance is the role's own failure mode
The OWASP Top 10 for LLM Applications names excessive agency and overreliance as distinct risks, and the distinction maps exactly onto this post. Excessive agency is a missing gate. Overreliance is a gate staffed by a human who has stopped looking.
Overreliance is measurable, which means it is manageable. 3 instruments cover most of it.
- Disagreement rate per reviewer per slice. A reviewer whose rejection rate sits far below the pool average is either your best reviewer or your most tired one, and time-per-item usually says which.
- Seeded known-bad items. Inject cases with known incorrect outputs at a low rate and measure the catch rate, the same way payment teams test their fraud queues.
- Second-level sampling. A senior reviewer audits a fraction of approved items, which gives the same statistical protection to the review layer that the review layer gives the model.
Rotation helps too. 4 hours of continuous review produces worse judgment than 2 blocks of 2, and a reviewer who also does the operational job stays calibrated on what correct looks like in a way a full-time reviewer slowly does not.
The regulators already treat it as a role
If the engineering argument does not move your steering committee, the compliance one will. The EU's regulatory framework for AI requires human oversight measures for high-risk systems, framed as minimizing risk through the way the system is overseen, not as a disclosure line in the terms of service.
Oversight you can demonstrate is oversight with artifacts: named reviewers, recorded verdicts, documented triggers for tightening, and evidence the reviewer could actually intervene. A 99.7% approval rate with 8-second review times demonstrates the opposite, in writing, with timestamps.
The building effective agents guidance from Anthropic points the same direction from the vendor side: it recommends adding complexity only when it demonstrably improves outcomes and keeping humans in a position to check agent work at meaningful checkpoints. A checkpoint nobody staffs is not meaningful, whatever the architecture diagram says.
The job description, written down
Everything above compresses into a 1-page document that should exist before the feature ships. If you cannot fill in a row, you have found the gap before production found it for you.
| Question | What a real answer looks like |
|---|---|
| Who reviews? | Named role with domain ownership, with cover for leave and peak |
| What do they see? | Decision, evidence spans, confidence, applied rule, effect-if-approved |
| What can they do? | Approve, correct, reject with reason code, escalate |
| What is gated vs sampled? | A written routing rule keyed on reversibility and value |
| What tightens the loop? | Named triggers: model change, policy change, new slice, disagreement spike |
| What loosens it? | Per-slice agreement thresholds held over a defined period, with sign-off |
| Where do verdicts go? | Golden set, eval slices, reason-code Pareto, threshold tuning |
| Who reviews the reviewer? | Seeded items plus second-level sampling, on a schedule |
Teams resist writing this document because it makes the true cost of the AI feature visible. That is precisely its value. A feature whose review cost makes it uneconomic should lose the argument in planning, not in month 4 with a burned-out reviewer and a compliance finding.
FAQ
Does every AI feature need a human review loop?
Every feature needs the routing decision made explicitly, and the honest answer for many is sampling only. A product-tagging workflow with cheap, reversible mistakes can run on evals plus a light audit. Money movement and customer-facing commitments need the gate.
Can the reviewer be another model?
A judge model is a grader, not a reviewer, and it belongs in the eval suite where its own agreement with humans is measured. Putting a second model in the approval seat moves the question of trust one layer back without answering it.
What approval rate should worry us?
Both tails. A rate near 100% for weeks means the gate is theater or the work has outgrown it, and either deserves a decision. A persistently low rate means the model is not ready for the traffic it is getting, and the gate is doing the model's job.
How do we keep reviewers from becoming a bottleneck at peak?
Capacity-plan for peak explicitly, and pre-agree the narrowing rule: which action types drop from gate to sample when volume doubles. Deciding that in October beats deciding it in the queue on Black Friday morning.
Should reviewers see the model's confidence score?
Yes, with care. Confidence is evidence for the reviewer, and hiding it wastes signal. The risk is anchoring, which the seeded known-bad items measure directly: if catch rates fall on high-confidence seeds, the display needs redesign, not removal.
Where does this live organizationally?
The workflow belongs to the domain owner, the tooling to engineering, and the thresholds to both, reviewed on a cadence. Splitting it differently produces either a review UI nobody uses or a policy nobody can implement.
References
- Google Engineering Practices - code review developer guide
- LangSmith - annotation queues
- Label Studio - open-source data labeling and review
- OWASP - Top 10 for Large Language Model Applications
- Google - People + AI Guidebook
- European Commission - regulatory framework for AI
- Anthropic - building effective agents