Skip to content
fitme·story
fitme·story
View as
Case studies
v7.10 · 3 min read · 2026-06-10

T10 — AI Golden-Set Eval Harness (and why the FitMe AI is deterministic, not generative)

T10 was spec'd as an "LLM golden-set eval harness — promptfoo-equivalent weekly run." Reading the actual code corrected the framing: the FitMe AI is a pure deterministic rule engine (InsightService), not generative. The live LLM path is gated behind an unset LLM_API_KEY. That makes a golden set strictly better than a promptfoo run — zero flake, no key, and it runs as a hard PR gate in the existing pytest suite rather than a flaky weekly advisory. 24 golden cases pin every InsightService rule, confidence band, and the escalate_to_llm threshold; ai-engine suite green at 60 passed / 1 skipped; a negative-control mutation proved the set catches AI-logic regressions by construction.

24
deterministic golden cases — 26/26 pass, run as a hard PR gate (not a flaky weekly advisory)
Reading the code reframed the spec: the FitMe AI is a pure deterministic rule engine (InsightService), not generative. That makes a golden set strictly better than promptfoo — zero flake, no LLM key. A negative-control mutation made exactly the matching case fail, proving the set catches AI-logic regressions by construction.
What do T1 / T2 / T3 mean?▾
T1Instrumented — from a ledger/commit
T2Declared — stated, not measured
T3Narrative — estimate from memory

T10 — AI Golden-Set Eval Harness

Status: shipped 2026-06-10. The first behavioral test coverage of FitMe's AI decision logic — the "biggest layer gap" per the test-coverage master plan.

The scoping surprise: the AI is deterministic, not generative

T10 was spec'd in the test-coverage master plan as an "LLM golden-set eval harness — promptfoo-equivalent weekly run." That framing assumed the AI is generative. It isn't.

ai-engine/app/services/insight_service.py::InsightService.generate() is a pure deterministic rule engine: given a segment + user_fields + cohort_totals, it emits population-level signals, a confidence score (coverage_ratio × 0.6 + cohort_signal × 0.4), and an escalate_to_llm boolean (confidence < 0.40). The generative LLM path is gated behind an unset LLM_API_KEY (requires a Data Processing Agreement) and is not the live behavior. The federated-cohort intelligence + on-device Foundation Models layer do the personalization; the server emits signals, not prose.

What shipped

  • ai-engine/tests/golden/insight_cases.jsonl — 24 golden cases [T1, instrumented] covering:
    • all 3 rules in each of the 4 segments (training / nutrition / recovery / stats) + their no-match cases;
    • confidence scoring (full-coverage-large-cohort → ≈1.0; sparse-tiny-cohort → escalates);
    • the escalate_to_llm < 0.40 boundary;
    • edge cases: empty user_fields (no div-by-zero), empty/saturated cohort_totals, unknown segment.
  • ai-engine/tests/test_golden_insights.py — a parametrized harness that runs each case through the real InsightService and asserts the behavioral contract (signals_contains / signals_excludes / signals_empty, confidence_min/max, escalate_to_llm, supporting_total_cohort_size), plus a coverage assertion (≥20 cases, all 4 segments present) and an explicit escalation-threshold pin.
  • Live-LLM eval scaffold — test_llm_eval_golden_subset_when_key_present skips cleanly when LLM_API_KEY is unset (the default), so the weekly promptfoo-equivalent run can land later without changing the harness shape.

Verification (all 2026-06-10, T1 — instrumented)

  • pytest tests/test_golden_insights.py → 26 passed, 1 skipped.
  • Full ai-engine suite → 60 passed, 1 skipped (34 pre-existing endpoint tests + the new behavioral coverage; no regression).
  • Negative control (the value proof): mutating one rule (cohort_muscle_gain_foundation_common → a typo) made exactly the corresponding golden case fail — the set catches AI-logic regressions by construction, not by rubber-stamp.

Why this matters

Before T10, the 34 ai-engine tests covered the transport (auth, payload validation, fire-and-forget) but none asserted what the AI actually decides. A refactor could silently drop the deload_advised recovery signal or shift the escalation threshold and every test would still pass. T10 closes that: the AI's decision behavior is now pinned, deterministic, and gated at PR time — the highest-value AI test work, whose only blocker (the Phase 2-bis prompt-set stabilizing) was cleared when all four sub-experiments closed PASS on 2026-06-05.

Where to read more

Honest disclosures
  • •The harness only exercises the deterministic InsightService rule engine — the generative LLM path is gated behind an unset LLM_API_KEY (requires a Data Processing Agreement) and is NOT tested live; the live-LLM eval scaffold skips cleanly by design, so this proves the rule engine, not any prompt quality.
  • •The 24 golden cases pin behavioral contracts authored by a single practitioner from reading the code — they catch regressions against the current intended behavior (negative-control proven), not against an independently-validated ground truth of what the AI *should* recommend.
Kill criterion · not fired
  • Golden harness becomes flaky (non-deterministic) — would violate the "deterministic AI is gateable" premise → revert to advisory.
  • Golden set pins implementation detail rather than behavioral contract (brittle to benign refactors).