HADF Phase 2-bis — Cross-Sub-Exp Synthesis (3 Sub-Experiments)
Successor to HADF Phase 2 (slot 22b). Three pre-registered sub-experiments test orthogonal aspects of the HADF dispatch claim: Sub-exp 1 (cloud generalization, silhouette ≥ 0.5), Sub-exp 2 (cloud-vs-local separability, KS p < 0.01), Sub-exp 3 (Bedrock vs Anthropic-direct same-model routing, signature delta > 2.0). Sub-exp 3 is the falsification test for the central HADF claim. Skeleton authored 2026-05-27 as Block C pre-work; verdicts populate as each sub-exp closes (Sub-exp 1 ~2026-05-28, Sub-exp 2 ~2026-05-31, Sub-exp 3 ~2026-06-05). **CLOSED 2026-06-05 — all four sub-experiments PASS: Sub-exp 1 (silhouette 0.70), Sub-exp 2 (KS p≪0.01), Sub-exp 3 (signature_delta_ratio 2.89 > 2.0 — the falsification test SURVIVES), Sub-exp 1B (silhouette 0.98, anchor drift 0.19σ). Synthesis verdict: HADF dispatch claim CONFIRMED — streaming signatures distinguish substrate, provider, and routing layer. Scope honesty: this confirms the SENSING layer; the dispatch ACTING layer (does routing on signatures improve outcomes?) is unproven and gated on RQ4 (Phase 3B).**
What do T1 / T2 / T3 mean?▾
TLDR
Successor to HADF Phase 2 (cloud fingerprinting measurement, silhouette 0.5566 at k=5 over n=700, shipped 2026-05-01). Phase 2-bis tests three orthogonal aspects of the HADF dispatch claim via independent pre-registered sub-experiments:
- Sub-exp 1 — Cloud generalization: Does the Phase 2 single-endpoint signature reproduce across multiple cloud providers? (silhouette ≥ 0.5 with cluster count ≥ 3 across 9 endpoints, narrowed to 4 at smoke-fire)
- Sub-exp 2 — Cloud-vs-local separability: Can we distinguish Ollama-on-M2 from cloud endpoints by signature alone? (KS-distinguishability p < 0.01)
- Sub-exp 3 — Same-model routing (FALSIFICATION TEST): Does AWS Bedrock haiku-4-5 fingerprint differently from Anthropic-direct haiku-4-5? (signature_delta_ratio > 2.0; the central HADF dispatch claim stands or falls here)
Each sub-exp ships an independent verdict. Sub-exp 2 launches only if Sub-exp 1 passes; Sub-exp 3 only if Sub-exp 2 passes. The synthesis (this case study) combines all three verdicts into a confirmed / partially-supported / refuted disposition.
Status
COMPLETE — all four sub-experiments closed 2026-06-05, all PASS, HADF dispatch claim CONFIRMED. The skeleton (authored 2026-05-27, Phase E Day 6, before any data) was filled in mechanically at closure, exactly as designed — no after-the-fact narrative drift.
| Sub-exp | Verdict | Metric |
|---|---|---|
| 1 — cloud generalization | ✅ PASS | silhouette 0.70 |
| 2 — cloud-vs-local | ✅ PASS | KS p ≪ 0.01 |
| 3 — routing falsification | ✅ PASS | signature_delta_ratio 2.89 (> 2.0) |
| 1B — cross-window drift | ✅ PASS | silhouette 0.98, anchor drift 0.19σ |
The keystone result: the same model (claude-haiku-4-5) behind two routing layers — Anthropic-direct vs AWS Bedrock — yields distinguishable streaming signatures (Bedrock TTFT 1.468 s vs Anthropic-direct 0.868 s). The central HADF falsification test survives.
Scope honesty: this confirms the sensing layer — signatures are detectable and discriminative. It does not establish the acting layer (that routing on signatures improves a real dispatch outcome). Distinguishability ≠ actionability; that question is RQ4 (HADF Phase 3B), pre-registered separately. HADF Phase 3A (sensing/observability) is greenlit for activation; routing logic stays gated until RQ4.
Reference
- Source case study (FT2 upstream):
docs/case-studies/hadf-phase2bis-cross-sub-exp-synthesis-case-study.md - Spec:
docs/superpowers/specs/2026-05-11-hadf-phase2bis-replication-design.md - Predecessor: HADF Phase 2 case study
- Per-sub-exp preregs: Sub-exp 1 (LOCKED) / Sub-exp 2 (DRAFT pending B14 ceremony) / Sub-exp 3 (DRAFT pending B15 ceremony)
- External Audit #2 substrate target: 2026-06-12
Verdict logic preview (populated at synthesis time)
| Sub-exp 1 | Sub-exp 2 | Sub-exp 3 | Synthesis disposition |
|---|---|---|---|
| PASS | PASS | PASS (delta > 2.0) | HADF dispatch claim CONFIRMED |
| PASS | PASS | INCONCLUSIVE (1.0 ≤ delta ≤ 2.0) | HADF dispatch claim PARTIALLY SUPPORTED |
| PASS | PASS | REFUTED (delta < 1.0) | HADF dispatch claim REFUTED on routing axis |
| PASS | FAIL | n/a | HADF dispatch claim PARTIALLY SUPPORTED (cloud generalization only) |
| FAIL | n/a | n/a | HADF dispatch claim CANNOT be confirmed by this measurement |
Observed combination so far: Sub-exp 1 PASS / Sub-exp 2 PASS / Sub-exp 3 pending (gated on AWS Bedrock access). Synthesis disposition is held until Sub-exp 3 closes.
Sub-exp 2 — Results (closed 2026-06-02): PASS
Research question: Can we distinguish Ollama-on-M2 from cloud endpoints by signature alone? Verdict: PASS — local execution is distinguishable from both cloud anchors at p ≪ 0.01 on both streaming metrics. Hypothesis (RQ2) confirmed.
Collection ran 2026-05-30T04:50Z → 2026-06-02T02:00Z (18 fires, 1 endpoint llama3.2:3b on M2, 50 calls/fire), cost $0 (local inference). Verdict computed against the lock-intact pre-registration (sha256 d4ec4680…, lock sidecar verified) — pass criteria fixed before data landed: KS p ≤ 0.01, yield ≥ 250 (kill floor 200).
Pass criteria vs observed (all T1 — instrumented from raw .jsonl):
| Metric | Threshold | Observed | Pass |
|---|---|---|---|
| n_valid (Ollama) | ≥ 250 (kill floor 200) | 800 / 900 dispatched (88.9% valid) | ✅ |
| KS p — TTFT vs openai/gpt-4o-mini | ≤ 0.01 | KS 0.9362, p ≈ 0 (< 1e-300) | ✅ |
| KS p — TTFT vs anthropic/claude-haiku-4-5 | ≤ 0.01 | KS 0.9812, p ≈ 0 (< 1e-300) | ✅ |
| KS p — TPS vs openai/gpt-4o-mini | ≤ 0.01 | KS 0.7492, p = 7.39e-273 | ✅ |
| KS p — TPS vs anthropic/claude-haiku-4-5 | ≤ 0.01 | KS 0.9100, p = 9.88e-324 | ✅ |
Full distribution summary (Ollama n=800; each cloud anchor n=1300) [T1]:
| Endpoint | TTFT_s (min / p25 / med / p75 / max) | TPS (min / p25 / med / p75 / max) | TPS std |
|---|---|---|---|
| Ollama llama3.2:3b (M2, local) | 0.140 / 0.147 / 0.179 / 0.227 / 5.005 | 23.1 / 32.8 / 43.8 / 45.1 / 46.2 | 6.9 |
| openai/gpt-4o-mini (cloud) | 0.000 / 0.001 / 0.021 / 0.055 / 32.2 | 0.3 / 46.2 / 65.4 / 102.1 / 3984.2 | 277.2 |
| anthropic/claude-haiku-4-5 (cloud) | 0.594 / 0.909 / 1.099 / 1.417 / 34.7 | 6.3 / 66.4 / 83.3 / 152.9 / 27638.2 | 951.2 |
What the numbers say. Two structural features carry the separation:
- Time-to-first-token. Ollama's 0.179 s median sits cleanly between openai's near-instant 0.021 s (edge-served, no model-load latency) and anthropic's 1.099 s. The local 3B model has no network round-trip but a real prompt-eval step — landing it in its own latency band.
- Throughput variance. Ollama's tok/s is tightly clustered (std 6.9, almost the entire mass in 32–45 tok/s), while both cloud endpoints show extreme variance (openai std 277, anthropic std 951) from short-output tok/s inflation and multi-tenant scheduling jitter. A local M2 inference loop is consistent in a way cloud serving is not — and that consistency is itself a fingerprint.
Cloud-vs-local separability confirmed. Sub-exp 3 (the same-model routing falsification test) is unblocked on this axis; its launch is gated separately on AWS Bedrock access.
Source: verdict artifact
phase2bis-subexp2-verdict.json· locked preregpreregistration-phase2bis-subexp2.json. Independent re-check of the raw.jsonl+ verdict math is in scope for External Audit #2 (2026-06-12).
- •Skeleton authored 2026-05-27 (Phase E Day 6) as Block C task C16 pre-work; verdict cells populate as each sub-exp closes. The skeleton exists to ensure the synthesis is mechanical-not-creative at closure — every analytical section is pre-structured before data lands, eliminating after-the-fact narrative drift.
- •Sub-exp 1 launched 2026-05-25 with a narrowed 4-endpoint matrix (openai gpt-4o-mini + gpt-4o + anthropic claude-haiku-4-5 + claude-sonnet-4-6) instead of the originally-specified 9 endpoints. Operator decision 2026-05-25 per smoke-fire findings (Gemini reasoning-model TTFT distortion; placeholder API keys not minted for Vercel AI Gateway + Mistral + xAI). Sub-exp 1B queued for full 9-endpoint follow-up after key acquisition + reasoning-model investigation.
- •Sub-exp 2 and Sub-exp 3 prereg
verdict_thresholdscarry suggested values (KS p < 0.01 / delta_ratio > 2.0) — operator finalizes at each go/no-go ceremony. The TBD fields in §3.B and §3.C of this case study mirror those pending decisions. - •Sub-exp 3 is structurally different from Sub-exps 1+2: it is a FALSIFICATION test for the central HADF dispatch claim. Same model id (anthropic.claude-haiku-4-5) served by Anthropic-direct + AWS Bedrock; if signatures differ enough to clear within-provider noise floor → HADF dispatch premise holds; if indistinguishable → HADF refuted on this metric. The verdict_thresholds.fail_if_signature_delta_ratio_lt is the pre-registered binary against which the central HADF claim stands or falls.
- •External Audit #2 is scheduled 2026-06-12 and covers raw .jsonl data integrity + verdict scripts + this case study + anchor-drift check output. No independent reassessment has occurred yet.
- Any sub-exp tripped per-sub-exp kill criteria (n_valid floor, rate-limit cascade, mid-collection protocol change, wrapper preflight failures) — see per-sub-exp prereg JSONs
- Sub-exp 3 anchor drift > 2σ from Sub-exp 1 — invalidates routing-test comparison (halt + investigate before drawing routing conclusions)
- Sub-exp 3 signature_delta_ratio < 1.0 — HADF dispatch premise REFUTED on this metric (operator decides claim revision)
gated on 3-day collection window close ~2026-05-28Block B task B13.5 — verdict-script run executes silhouette + cluster count + threshold checkCLOSED 2026-06-02 — verdict PASS (n_valid 800; KS p ≪ 0.01 vs both cloud anchors)Block B task B14 done — Ollama local fingerprinting; §3.B + this showcase populatedgated on Sub-exp 2 verdict = PASSBlock B task B15 — Bedrock vs Anthropic-direct routing test; prereg fill-in ready at #507gated on Sub-exp 3 closure ~2026-06-04Block C — populates §3.A, §3.B, §3.C verdicts + §4 anchor-drift + §5 overall HADF claim statusscheduled 2026-06-12 per audit substrateIndependent operator reviews raw .jsonl + verdict scripts + this case study