# v1.8 Prereg "v1.9 LongMemEval Cross-Validation" Pre-Commitment — Retracted

**Effective:** 2026-05-09 (shipped in v1.8.1)
**Author:** Keith So
**Per:** `docs/RETRACTION.md` honest-reporting discipline + the v1.7.9 pre-emptive retraction precedent.

**This release does not re-assert the retracted −10pp magnitude.** Per `docs/RETRACTION.md`, mechanism characterisation only.

## What was pre-committed

The v1.8.0 pre-registration (`docs/evals/2026-05-09-v1.8.0-adversarial-eval-prereg.md`, "Pre-committed v1.9 direction" subsection, drafted BEFORE v1.8 ran) committed:

> v1.9 will run the dlPFC goal-stack mechanism on the LongMemEval R@5 corpus as a cross-validation of the mechanism on a fundamentally different benchmark.

That pre-commitment is **RETRACTED** publicly in v1.8.1.

## Why the pre-commitment is retracted

The v1.9 design space, on the LongMemEval corpus and the existing harness, is structurally too narrow for the goal-stack mechanism to fire without substantial re-architecture. Outside-voice review on **two iterations of the v1.9 plan** (v1 and v2 — see `docs/plans/2026-05-09-v1.9.0-longmemeval-cross-validation.md` for the v2 design) identified six concrete structural barriers, each independently confirmed by source-reading.

### Structural barrier 1 — wrong harness function

The canonical LongMemEval harness `benchmarks/longmemeval/retrieve_inprocess.mjs:15` imports `hybridSearch` directly from `dist/search.js` and calls it at line 68. The dlPFC `applyGoalStackBoost` mechanism is invoked ONLY from `src/cli.ts::cmdRecall` (line 1008), `src/api.ts::recall` (line 467), and `src/mcp/server.ts` (line 497). It is NOT invoked from `hybridSearch` or `physicsSearch`. The harness as shipped structurally bypasses the boost path.

### Structural barrier 2 — auto-tag namespace mismatch

LongMemEval ingest (`benchmarks/longmemeval/ingest_direct.py:139`) writes session-tags as `[sid]` plus `f"date:{date}"` — that is, the LongMemEval session UUID and the session date. Zero content-derived tokens. The boost match predicate is `goalsByTag.has(memoryTag)` exact string equality on `entry.tags` (`src/goals.ts:265` + `src/goals.ts:302`). Porter-stemmed question keywords (e.g. `meet`, `schedul`, `user`) cannot match `["session_47e9a2", "date:2024-04-15"]` under exact equality. Boost firing rate is structurally 0 on this corpus.

### Structural barrier 3 — pushGoal API field mismatch

The v1.9 v2 plan invoked `pushGoal({tenantId, sessionId, tag: stem})`. The actual `PushGoalOpts` shape (`src/goals.ts:69-83`) requires `goalName`, not `tag`. The match predicate uses `goalName` against `entry.tags` (`src/goals.ts:265`). v2's call would have TypeScript-errored or silently ignored.

### Structural barrier 4 — pushGoal depth cap interaction

`MAX_ACTIVE_GOAL_DEPTH = 3` (`src/goals.ts:117`). v2 plan pushed top-3 stems per question; the third push would suspend the first stem before recall fired. `getActiveGoalsWithDb` filters `WHERE status = 'active'`, excluding suspended goals from the boost. Net effect: only stems[1] and stems[2] would have participated.

### Structural barrier 5 — cumulative-null trigger AND clause unreachable

v2 pre-committed `SAME ≥ 495/500 AND boost-firing-rate ≥ 50/500 → escalate RETRACTION.md`. Per Barrier 2, boost-firing-rate is structurally 0/500. The AND clause is mathematically unreachable. The trigger as designed never fires; the cumulative-null escalation never appends.

### Structural barrier 6 — workload-validity gate ceremonial

v2 set the C1 sanity gate at `R@5 ∈ [0.55, 0.90]`, centred on the historical 74% baseline. PASS is structurally guaranteed by the harness alone. The gate does not discriminate the mechanism; it is procedural-not-discriminating, and acknowledged as such in the v2 plan.

## Why retraction over re-architecture

Three options were available after outside voice round 2:

- **Option A** — modify `ingest_direct.py` to add content-keyword auto-tags + re-ingest 940 LongMemEval sessions. ~4-6 hours wall-time + new C0 baseline + risk of ingest-schema-driven R@5 drift confounding the comparison.
- **Option B** — modify `retrieve_inprocess.mjs` to call `api.recall` instead of `hybridSearch`. ~2-3 hours wall-time + accept that the new harness path may produce a different R@5 than historical 74% + re-validate sanity gate on the new path.
- **Option C** — retract the pre-commitment publicly and propose an alternative direction.

Per `CLAUDE.md` "Root Cause Over Patches" — the structural fix is **NOT** a workaround within the LongMemEval scope. The original v1.8 pre-commitment was made before the source code that constrains v1.9's design had been read. Forcing a workable v1.9 via option A or B is a patch over a design-mismatch: we'd be testing the goal-stack mechanism on a *modified* LongMemEval (different ingest schema or different harness call shape), not on LongMemEval-as-shipped. The cross-validation framing fails either way.

Per the v1.7.9 retraction precedent (pre-emptive retraction on cumulative evidence), option C is the principled call.

## What this retraction does

1. **The v1.8 prereg's v1.9 LongMemEval cross-validation pre-commitment is retracted publicly.** The v1.9.0 release slot is closed. Future eval releases use fresh pre-registration with source-validated design assumptions.

2. **`docs/RETRACTION.md` is updated** with two new subsections:
   - "## v1.9 pre-commitment retraction" — this retraction.
   - "## Mechanism-effect status (cumulative null escalation)" — pre-committed in the v2 plan as a trigger; fires now because the cumulative null evidence (v1.7.5 SANITY_FAIL, v1.7.6 B*=NULL, v1.7.7 SANITY_FAIL, v1.8.0 SAME=20/20) PLUS the v1.9 untestability collectively argue that the dlPFC goal-stack mechanism does not produce a detectable behavioural effect on the workloads we have tested. The mechanism remains shipped from v1.7.4. No specific effect is currently claimed.

3. **The v1.10 pre-commitment (per v1.9 plan v2) is retracted alongside.** "Iterate goal-tag mapping" was downstream of the v1.9 design that is now retracted.

4. **A new pre-registration discipline rule is committed:** **No future eval pre-commitment without (a) source-read of the code paths the design depends on, AND (b) a 1-question dry-run wired through the actual mechanism path that confirms the mechanism FIRES before pre-reg locks.** This rule lives at the top of `docs/RETRACTION.md` as a methodology guard.

## What this retraction does NOT do

- Does NOT retract the dlPFC goal-stack mechanism. The code remains shipped from v1.7.4: `pushGoal` / `completeGoal` hooks, `--use-goal-stack` CLI flag (sequential-learning runner), `applyGoalStackBoost` helper, MCP `hippo_recall { session_id }` boost, HTTP `GET /v1/memories?session_id=` boost. All shipped. The mechanism's CODE is fine; what is retracted is the *claim that this mechanism can be cross-validated on the LongMemEval corpus as currently scoped*.
- Does NOT retract v1.7.5/6/7/8 results. Those stand. The cumulative null escalation builds on them; it does not retract them.
- Does NOT pre-commit a new v1.9.x or v1.10.x eval direction. The lesson is: pre-commit the *rule* (source-read + dry-run before pre-reg), not the next eval target. The next eval target will be drafted in a separate plan, with the rule applied.

## Alternative directions for future releases (NOT pre-committed; for context)

When a new eval direction is decided, candidate workloads where the goal-stack mechanism is more likely to be testable:

1. **Synthesised multi-turn conversation eval** where session-scoped goals are *natural* features of the data (not pre-tagged metadata). Per-session goals could be the topic-of-conversation, set explicitly at conversation creation. Boost mechanism fires when subsequent recalls share the same topic.
2. **Mechanism-removal A/B telemetry** — ship hippo with goal-stack OFF as a config option. Observe in real-world usage (hippo's own dogfooding) whether the goal-stack ON / OFF produces user-visible differences. No pre-registered eval; pure observability.
3. **Infrastructure-only release** with no eval. Ship the deferred items (P2-1 Float64Array micro-opt from v1.7.10 deferral; bug-fix queue) and skip mechanism-validation for one release.

None of these are pre-committed by this retraction. Future plans pick one + apply the new methodology rule.

## Audit trail

- Plan v1: `docs/plans/2026-05-09-v1.9.0-longmemeval-cross-validation.md` was overwritten by v2; v1's content lives in v1's outside-voice round-1 findings (in this session's transcript).
- Plan v2: `docs/plans/2026-05-09-v1.9.0-longmemeval-cross-validation.md` (current file at HEAD; preserved as historical-retracted record).
- Outside voice round 1 (on v1): 28 findings, 4 architectural P0s.
- Outside voice round 2 (on v2): 21 findings, 6 P0s including 3 NEW structural P0s the v1 review missed.
- Total findings across both rounds: 49 (with overlap).
- v1.7.9 retraction precedent: `docs/RETRACTION.md` v1.7.9 subsection.

## Related documents

- `CHANGELOG.md` v1.8.1 entry (top of file).
- `docs/RETRACTION.md` (updated with two new subsections).
- `README.md` "What's new in v1.8.1" section.
- `ROADMAP-RESEARCH.md` B3 Status update.
- `TODOS.md` updated B3 line.
- `docs/plans/2026-05-09-v1.9.0-longmemeval-cross-validation.md` (preserved historical-retracted plan v2).
