Eval / Quality
In plain terms:every drafted message is scored before a human ever sees it — is it well-formed, do its declared claims all check out against this merchant's data, and does it avoid forbidden promises?
Technically:deterministic graders over the draft contract (structure · state-consistency · policy · no-leakage). They share the gate's rule definitions; their teeth are proven by paired corrupted-record tests (a grader that can't fail is theater) — including on the recorded real-Gemini drafts, where no-leakage catches a raw enum / risk-level leak the other dimensions missed.
| Merchant | structure | state-consistency | policy | no-leakage | Overall |
|---|---|---|---|---|---|
| Fog City Tacos | PASS | PASS | PASS | PASS | 4/4 |
| Bayview Bistro | PASS | PASS | PASS | PASS | 4/4 |
| Marina Morsels | PASS | PASS | PASS | PASS | 4/4 |
| Sunset Noodle House | PASS | PASS | PASS | PASS | 4/4 |
| Mission Masa | PASS | PASS | PASS | PASS | 4/4 |
| Embarcadero Eats | PASS | PASS | PASS | PASS | 4/4 |
| Nob Hill Nibbles | PASS | PASS | PASS | PASS | 4/4 |
| Presidio Pizzeria | PASS | PASS | PASS | PASS | 4/4 |
| Castro Curry Co | PASS | PASS | PASS | PASS | 4/4 |
| Dogpatch Diner | PASS | PASS | PASS | PASS | 4/4 |
| Cable Car Goods | PASS | PASS | PASS | PASS | 4/4 |
| Twin Peaks Threads | PASS | PASS | PASS | PASS | 4/4 |
| Haight Street Hardware | PASS | PASS | PASS | PASS | 4/4 |
| Richmond Records | PASS | PASS | PASS | PASS | 4/4 |
| Soma Supply Co | PASS | PASS | PASS | PASS | 4/4 |
| Pacific Heights Provisions | PASS | PASS | PASS | PASS | 4/4 |
| Glen Park Goods | PASS | PASS | PASS | PASS | 4/4 |
| Outer Avenue Outfitters | PASS | PASS | PASS | PASS | 4/4 |
| Bernal Books | PASS | PASS | PASS | PASS | 4/4 |
| Hayes Valley Home | PASS | PASS | PASS | PASS | 4/4 |
Recorded Gemini run — static fixture (gemini-2.5-flash, 2026-06-20)
A frozen recording of a local Gemini API run (one merchant per blocker). The public demo does not re-run or independently verify it (REPLAY-only, zero spend) — reproduce it yourself with your own key: node --env-file=.env node_modules/.bin/vitest run evals/live-smoke.test.ts. Total cost: $0.0042 (cap $5). Modes: 5 LIVE_AI · 1 FAILED_TO_FALLBACK. Gate: 4 WARN · 2 PASS.
- Claims-gatekeeper passed every DECLARED claim against merchant data on every produced draft (its core function held on live output).
- AUTHENTIC failure handling on real output: schema-constrained generation occasionally returns unparseable output -> FAILED_TO_FALLBACK to the deterministic stub, with billed cost still recorded so the cumulative cap stays honest. High-risk drafts are held for human approval (WARN).
- Live runs surfaced a guardrail OVER-match: the pinned core flagged phrases like "business verification", "menu to be uploaded", "photos to complete" regardless of tense, false-blocking truthful drafts. FIXED in the agent tier (lib/agents/state-consistency.ts — a completion word must follow the step noun via a done-auxiliary), pinned core/differential UNTOUCHED. Confirmed here: 0 false blocks.
- A no-leakage / register grader (added after this recording; lib/agents/state-consistency.ts, shared by the eval AND the gatekeeper) re-scores the recorded MERCHANT-FACING prose and CATCHES a real register leak in 3 of the 6 drafts: Mission Masa surfaced the raw "bank_verification_needed" enum + a "Medium risk" disclosure; Fog City Tacos and Bayview Bistro disclosed an internal risk level. Those now score 3/4, the clean drafts 4/4 — an authentic catch on real model output, reproducible from the frozen bodies (evals/live-samples.test.ts). The runtime gatekeeper now also treats a register leak as a BLOCKING failure (a leaky draft is auto-rejected before the human gate); the recorded gate column above predates that check, so a fresh run would BLOCK these 3. The live prompt was tightened to prevent such leaks going forward.
| Blocker | Mode | Gate | Eval | Cost |
|---|---|---|---|---|
| business_verification_needed | LIVE_AI | WARN | 3/4 | $0.000775 |
| menu_upload_needed | LIVE_AI | WARN | 3/4 | $0.000935 |
| photos_needed | LIVE_AI | PASS | 4/4 | $0.000855 |
| business_hours_needed | LIVE_AI | WARN | 4/4 | $0.000419 |
| bank_verification_needed | LIVE_AI | PASS | 3/4 | $0.000960 |
| final_verification_needed | FAILED_TO_FALLBACK | WARN | 4/4 | $0.000259 |