"The classifier is only half the system. The human is the other half — and right now, they have no interface."
A shipped human review & evaluation console for the Food Not Food classifier. Uncertainty-first triage, calibrated confidence display, RLHF comparison mode, and an eval dashboard that closes the feedback loop between annotator and model.
In the Loop is the third layer of a deliberate AI product stack — each project building on and closing the loop of the one before it.
The design system foundation
Enterprise UX across 35+ flows at Primerica. Established the pattern library and component thinking that everything else inherits from.
The model that needs a reviewer
Fine-tuned DistilBERT, F1=1.0, deployed on Hugging Face. But "100% accuracy" hides the cases where confidence is low — the model knows it's uncertain.
The interface that closes the loop
The human review console. Uncertainty-first triage, calibrated confidence, RLHF comparison, eval dashboard. 19 components. 122 tests. Shipped.
The classifier surfaces predictions. But without a review interface, a human annotator has no way to triage, correct, compare, or measure anything. Three specific gaps drove the entire design.
Uncertainty has no visual signal
confidence: 0.51 → displayed same as confidence: 0.99
A 51% prediction and a 99% prediction look identical to a reviewer. High-uncertainty items aren't surfaced first — they're buried in a flat list.
Uncertainty-first sort with calibrated confidence bars and explicit tier labeling (High · Moderate · Low confidence). Reviewers see risk first.
Corrections vanish — no RLHF loop
correction: null → model never sees human signal
Human corrections to classifier output are captured nowhere. The model has no mechanism to learn from reviewer disagreements.
RLHF comparison mode: side-by-side model output vs. human label. Disagreements are logged with rationale fields ready for fine-tune export.
No eval layer — quality is invisible
accuracy: unknown → no dashboard, no trend, no metric
Post-deployment, there's no way to know if the model is degrading, improving, or staying flat. There's no surface for eval metrics.
Eval dashboard with agreement rate, confidence distribution, disagreement trend, and per-category breakdown. The loop is now closed.
The console is intentionally narrow. Each screen addresses exactly one of the three problems — no more, no less.
Review Queue
TriageUncertainty-first list view. Items sorted by classifier confidence ascending — the most ambiguous predictions surface first. Each row shows text, prediction, confidence tier, and a one-click correction interface.
Reviewer spends time where model is least certain
RLHF Comparison
DisagreementSide-by-side view: model prediction on the left, human label on the right. Disagreements are highlighted and logged with a rationale field. Data is structured for fine-tune export.
Corrections are captured and attributed, not lost
Eval Dashboard
MetricsLive evaluation metrics pulled from the review session. Agreement rate, confidence distribution chart, disagreement trend over time, and per-category breakdown. The loop is now closed.
Model quality is now visible and measurable
Item Detail
Deep reviewFull context view for a single classification. Shows the source text, model confidence breakdown by label, annotation history, and the rationale thread from previous reviewers.
Reviewer has full context for every ambiguous call
Every non-obvious choice is documented here — not to show process, but because the reasoning is part of the design.
Uncertainty-first sort is the default, not an option
Why not let reviewers choose their own sort order?
If uncertainty-first is optional, reviewers will default to the list order (FIFO) and never reach the ambiguous items — defeating the entire purpose of the review console. Opinionated defaults produce better annotation quality than flexible ones.
Power users who want FIFO or alphabetical order have to fight the default. Acceptable: the primary user is an annotator optimizing for model improvement, not personal preference.
Confidence is shown as a tier + bar, not a raw float
Why not show "63.2%" directly?
Raw floats invite false precision. A non-technical annotator will treat 63.2% and 64.1% as meaningfully different when they are statistically equivalent. Tiers (High / Moderate / Low) communicate the signal that actually matters: how much does the model doubt itself?
Researchers and technical users may want the raw number. It's available on the Item Detail screen — one click deeper, preserving the tier simplicity in the queue view.
Inline correction — no modal
Why not open a dialog for label correction?
Annotators review hundreds of items per session. A modal adds two interaction steps (open, confirm) for every correction. At 200 items, that's 400 extra clicks. Inline editing with keyboard shortcuts (Y/N to confirm/reject) keeps the flow unbroken.
Inline editing increases the risk of accidental corrections. Mitigated by an undo action available for 5 seconds after any correction, and a session review step before final submission.
Rationale fields are optional but encouraged
Why not require a rationale for every disagreement?
Required rationale creates annotation fatigue and leads to low-quality filler text ("wrong" / "disagree") that adds noise to the training data. Optional rationale with clear affordance produces fewer but higher-quality annotations from motivated reviewers.
Many corrections will have no rationale, reducing the interpretability of the RLHF signal. Mitigation: corrections with rationale are weighted higher in the export schema.
Export format is JSON, not CSV
Why not export to spreadsheet?
The downstream consumer is a fine-tuning pipeline, not a spreadsheet. JSON preserves the nested structure (original prediction, human correction, rationale, confidence tier, timestamp) without lossy flattening. Hugging Face Datasets accepts JSON natively.
Non-technical stakeholders who want to read the data need a viewer. A CSV export button is a future nice-to-have — the primary format is optimized for the model, not the spreadsheet.
Dashboard metrics are session-scoped, not global
Why not show all-time accuracy numbers?
Global accuracy numbers are stable and boring — they change slowly and don't motivate reviewers. Session-scoped metrics give real-time feedback on the current annotation batch: "you've reviewed 43 items this session, agreement rate is 87%."
Stakeholders who need longitudinal model performance have to aggregate across sessions. The export includes session metadata for this purpose.
50 seed items, hand-curated, not random
Why not use the model's own uncertain predictions as seed data?
Model-generated uncertainty lists are biased toward the model's existing failure modes. Hand-curation allows deliberate coverage of edge cases (ambiguous inputs, cultural food references, domain-specific terms) that the model wouldn't flag as uncertain because it doesn't know it doesn't know them.
50 items is a small seed. It's enough to demonstrate the review flow and generate meaningful session metrics, but not enough to produce a statistically significant RLHF training signal. Scale is a v2 concern.
Every review item flows through a single data structure that serves the queue, the RLHF export, and the eval dashboard simultaneously.
Review item schema
{
"id": "item-001",
"text": "I had an incredible bowl of...",
"model_prediction": "food",
"model_confidence": 0.51,
"confidence_tier": "low",
"human_label": null,
"correction": null,
"rationale": null,
"reviewed_at": null,
"session_id": "sess-2026-06"
}
confidence_tier
Derived from model_confidence: Low (<0.65), Moderate (0.65–0.85), High (>0.85). Drives sort order and visual tier pill in the queue.
correction + rationale
Null until reviewer acts. When populated, the item enters the RLHF export and the session disagreement metric. The presence of a rationale increases the item's weight in the fine-tune schema.
Confidence tier distribution (seed data)
| Tier | Confidence range | Seed count | % of queue |
|---|---|---|---|
| Low | < 0.65 | 18 | 36% |
| Moderate | 0.65 – 0.85 | 21 | 42% |
| High | > 0.85 | 11 | 22% |
| Total | — | 50 | 100% |
Seed data intentionally skews toward Low and Moderate tiers to maximize the value of human review — High-confidence items are unlikely to need correction.
These are named patterns from the emerging field of AI-UX — applied deliberately, not incidentally, in the review console.
| Pattern | Where it appears | Why it matters here |
|---|---|---|
| Uncertainty surfacing | Review queue sort order, confidence tier pills | Makes model doubt visible — reviewers act where it matters most |
| Calibrated confidence | Tier labels (High / Moderate / Low) instead of raw floats | Prevents false precision; communicates actionable signal |
| Human-in-the-loop Differentiator | The entire console — every screen | Makes human oversight a first-class product feature, not an afterthought |
| Preference capture | RLHF comparison mode, rationale fields | Structured disagreement creates training signal from human judgment |
| Progressive disclosure | Tier in queue → raw confidence in item detail | Shows simplified signal in the flow; full data one click deeper |
| Eval as UX | Dashboard with agreement rate, trend, distribution | Eval metrics aren't internal tooling — they're the product's feedback loop |
| Keyboard-first interaction | J/K navigation, Y/N correction, Space to confirm | High-volume annotation requires zero-friction input |
The build followed a strict sequence: data layer first, components second, screens third, eval last. No screen was started until its components passed tests.
| # | Output | Why this order | Tests |
|---|---|---|---|
| 1 | Scaffold + dark theme | Unblocks all visual work | — |
| 2 | Seed data + comparison pairs | All components have realistic data to render against | — |
| 3 | reviewReducer + ReviewContext | State logic verified before any UI consumes it | 12 |
| 4 | Sort, filter, eval utilities | Pure functions, fully tested, consumed by Tasks 5–9 | 30 |
| 5 | App shell + nav tabs | Navigation frame before content | 8 |
| 6 | Review Queue | Entry point — first thing a user sees | 19 |
| 7 | Single-Item Review + keyboard | Core interaction — where failure-mode UX lives | 23 |
| 8 | Eval Dashboard | Depends on reviewed items existing in state | 15 |
| 9 | Comparison Mode | Highest-signal screen — last because it depends on the full app being stable | 15 |
| Total | 122 / 122 |
Review the seed queue, make corrections, compare model vs. human labels in the RLHF view, and check session metrics on the eval dashboard.
Use J / K to navigate · Y / N to correct · Space to confirm
Open Live Demo →