Evalboard. Synthetic dataset · real model runs

The eval suite behind a sentiment feature.

An HR product wants to auto-label employee feedback. Which model, which prompt, at what cost? This workbench holds the answer the way a PM should hold it: a 200-comment gold dataset with deliberately hard slices, 3 prompt versions, 2 models, all 6 configurations actually run against the Claude API, and a decision log that says what shipped and why. Every number on this page comes from those runs.

200 gold labels · 4 classes 6 real eval runs · 48 API calls Hard slices: sarcasm, mixed, Hinglish, short Built with Claude Code

Accuracy and macro F1 across all six configurations

Prompt iterations (v1 naive, v2 adds label definitions and rules, v3 adds few-shot examples and sarcasm/code-switch handling) matter almost as much as model choice. The dotted ring marks the shipped configuration.

Haiku 4.5Sonnet 5

The decision chart: accuracy against cost

Cost per 1,000 comments (from real run telemetry) against accuracy. The gap between the two models closes as the prompt improves, which is the whole story.

Haiku 4.5Sonnet 5labels: prompt version

Drill into one configuration

Per-class F1, the confusion matrix, and slice accuracy for the selected run. The failure frontier is not sarcasm (both models solved it); it is the boundary between mixed and neutral, and short low-signal comments.

Per-class F1

Confusion matrix gold rows × predicted columns

Accuracy by slice

Failure explorer

Every miss for the selected configuration, with the model's prediction against gold. Click a row to see what all six configs predicted for that comment, plus review notes on the interesting ones. Two of the shipped config's seven misses are arguably label noise, which is exactly what a failure review is for.

CommentGoldPredictedSlice

Decision log: what shipped and why

  1. v1 told us the baseline was already high. Modern models classify plain-polarity HR comments near ceiling out of the box (Haiku 93.0%, Sonnet 96.0%). The naive prompt's misses concentrated in the mixed class (F1 0.74 on Haiku) and short comments, not in sarcasm, which both models handled at 100% from the first run. The eval killed a planned "sarcasm handling" workstream before it started.
  2. v2 taught us that definitions alone do not move the needle. Adding label definitions and tie-break rules without examples was flat to slightly negative on Haiku (mixed F1 dropped to 0.71). Rules describe the boundary; they do not demonstrate it.
  3. v3 closed the gap with few-shot examples. Four worked examples lifted Haiku to 96.5% accuracy and mixed F1 from 0.74 to 0.90, within half a point of Sonnet at roughly one fifth of the cost.
  4. The remaining frontier is annotation, not modeling. Of the shipped config's 7 misses, at least 2 are comments where the gold label itself is arguable. The next cheapest accuracy gain is tightening the labeling guideline for mixed versus neutral, then re-adjudicating that slice, not a bigger model.
Shipped: Haiku 4.5 with prompt v3. 96.5% accuracy, 0.94 macro F1, $1.14 per 1,000 comments versus $5.34 for Sonnet 5 at 97.0%. At feedback-pipeline volumes, the half-point of accuracy does not buy back a 4.7x unit cost, and the residual errors sit on a class boundary that annotation work fixes more cheaply than model spend. Guardrail: re-run this suite on every prompt or model change; alert if mixed F1 drops below 0.85.

Methodology, honestly

Dataset

200 synthetic employee-survey comments written for this demo, labeled with a 4-class scheme (positive, negative, neutral, mixed) and sliced into deliberately hard subsets: sarcasm (18), mixed-signal (19), Hinglish code-switching (22), very short (15), plain (126). No real employee data anywhere.

Runs

All 6 configurations (Haiku 4.5 and Sonnet 5 × prompt v1/v2/v3) were actually executed via the Claude CLI in August 2026, in batches of 25 comments, 48 API calls, $3.96 total spend. Predictions, latency, and cost were captured per batch and are rendered here unmodified. Cost figures include the CLI harness overhead, so treat them as realistic upper bounds rather than raw API list price.

Metrics

Accuracy, per-class precision/recall/F1, macro F1, confusion matrices, and slice accuracy are computed from those stored predictions by a ~90-line Python script. Nothing is estimated or extrapolated except cost and latency per 1,000 comments, which scale the measured per-batch telemetry linearly.

What this is not

Not a benchmark of the models in general, and not production data. It is a working demonstration of the eval discipline I bring as a PM: gold data with hard slices, versioned prompts, cost as a first-class metric, failure review, and a written ship decision.