# MCP Hang RCA — Why Priority Forge MCP calls occasionally wedge

**Status:** Investigated 2026-04-13. Root cause **localized to the Claude Code MCP client stdio path** (our side is fully exonerated by measurement). **Resolved 2026-04-13 (later same day) by migrating Claude Code to HTTP-transport MCP** — see §Resolution below. Stdio remains available as a fallback for emergencies.

**TL;DR (original):** Our proxy is fine. Our backend is fine. The hang is in Claude Code's MCP stdio read loop — it occasionally stops draining the socket, leaving the proxy's response buffered indefinitely. We can't fix Claude Code from outside. The REST API is the reliable fallback and is already wired into the mandatory protocol in `~/.claude/CLAUDE.md`.

**TL;DR (post-fix):** Claude Code now connects to the backend's `http://127.0.0.1:3456/mcp` endpoint directly via `type: "http"` in `.mcp.json` — same transport Cursor and Droid have always used. The stdio proxy is no longer in the request path, so the entire wedge class (stdio framing auto-detect, client-side stdio state machine, orphan proxies) is structurally eliminated. REST fallback stays in place as belt-and-suspenders.

---

## Symptoms

- An MCP tool call (e.g. `mcp__priority-forge__complete_task`) appears to hang for minutes.
- The Claude Code UI shows `Running... ↓ 0 tokens` with the elapsed time counter climbing.
- User interrupts (Ctrl+C) — when they do, the Claude Code session shows "tool call rejected."
- The backend's task state reflects that the operation COMPLETED (the request was processed normally) — so the work happened, only the *response* was never delivered back to the agent.
- Observed at least twice today (2026-04-13), both for `complete_task`, both resolved by user interrupting and failing over to REST.

## What we have proven is NOT the cause

### Backend is healthy
```
curl -s -m 2 http://127.0.0.1:3456/health  → {"status":"ok",...} in 8ms
systemd priority-forge-backend.service  → active (running) since Apr 9, 0% CPU, 0.2% mem
```

### Backend processes the MCP request normally
Direct curl bypassing the proxy:
```
curl -X POST http://127.0.0.1:3456/mcp -d '{"jsonrpc":"2.0","id":1,"method":"tools/call",...}'
→ valid JSON-RPC response in 17ms, matching id=1, all fields populated
```

### REST is fully functional at the same moment MCP hangs
```
curl -X POST http://127.0.0.1:3456/tasks/PFORGE-CONC-001/complete → 9ms, task completed
```

### The proxy is alive and doing I/O
From `/proc/<proxy-pid>/io` during a stable session:
```
rchar: 173802       ← proxy has read 170 KB from stdin
wchar: 129547       ← proxy has written 127 KB to stdout
```
That's sustained bidirectional traffic, not a dead process.

### The proxy has no stale orphans
After yesterday's proxy fix (signal handlers + watchdog reaper), `pgrep -f mcp-proxy.js` shows only live proxies with valid parents (`ppid != 1`).

### The proxy doesn't have a socket backpressure problem
```
ss -tnpx | grep <socket>
u_str ESTAB  recv_q=0  send_q=0  ...proxy fd=0
u_str ESTAB  recv_q=0  send_q=0  ...claude fd=19
```
Both directions idle, queues empty. No kernel-buffered data waiting.

## What we have localized the bug to

The stdio socket between the proxy and Claude Code has **TWO active endpoints**:
- **proxy side** writes responses — instrumented, known to work (wchar grows over time)
- **claude side** reads responses — ***NOT instrumented, not our code***

When the hang occurs:
1. Backend returns the response in ~20 ms.
2. Proxy's `forwardToHTTP` → `res.on('end')` → `writeToStdout(body)` runs. `process.stdout.write()` returns without error.
3. From the proxy's perspective: **the response was delivered.**
4. Claude Code never surfaces the response; elapsed time keeps climbing.

This means either:
- **(a)** the proxy wrote bytes that Claude Code is not reading, OR
- **(b)** Claude Code read the bytes but its MCP client state machine dropped them.

We cannot inspect Claude Code's internals. But we have eliminated the proxy and backend, so the bug lives in (a) or (b) — both inside Claude Code's process.

## Likely root causes (ranked by evidence)

### 1. Claude Code MCP client state-machine wedge (most likely)
The MCP JSON-RPC protocol matches responses to requests by `id` field. If Claude Code's client loses track of a pending-request id (e.g. a race in its own Promise bookkeeping, an exception in a middleware handler, a dropped callback on a retry path), the response arrives but has nobody waiting on it.

**Evidence:** the hang resolves ONLY by restarting Claude Code (not by restarting our proxy or backend). That's a fingerprint of in-process state, not transport state.

**Fix:** on the Claude Code side. Needs their team or a client-side patch that auto-detects stalled requests and retries/reconnects.

### 2. Framing-mode mis-detection in the proxy (possible)
The proxy auto-detects stdio framing from the first byte of the first message:
```js
framingMode = inputBuffer[0] === 0x7B /* '{' */ ? 'newline' : 'content-length';
```
Set once, used for ALL subsequent writes. If the first byte is anything other than `{`, it commits to Content-Length framing. A modern Claude Code expecting newline-delimited JSON would then fail to parse every response → wait forever.

**Evidence:** theoretical. No direct proof the first byte was mis-detected. But: the detection is a single-point, session-lifetime decision with no re-negotiation — exactly the kind of pattern that breaks silently.

**Fix:** change the proxy to always write BOTH framings, OR have Claude Code communicate its preferred framing explicitly (env var, CLI arg, initialize request field).

### 3. Unrelated libuv / Node event loop stall (unlikely)
A Node.js bug or GC pause could freeze the event loop. Our proxy would appear stuck.

**Evidence:** none. Context-switch counts look normal. I/O stats show consistent activity.

**Fix:** n/a unless this turns out to be the cause.

## What yesterday's fix (signal handlers + watchdog reaper) DOES and DOES NOT solve

**Solves:**
- Stale proxies from prior Claude Code sessions that got orphaned on Ctrl+C / terminal close (the reaper kills them after 60min).
- Current session's proxy cleanly exits on SIGTERM/SIGHUP/SIGINT instead of hanging.

**Does NOT solve:**
- The stdio wedge described above. That's in Claude Code's process, not the proxy's lifecycle.

## Immediate mitigations (all already in place)

1. **REST API fallback.** `~/.claude/CLAUDE.md` protocol section mandates using REST when MCP disconnects mid-session. REST is wire-equivalent for ML signal — same handlers, same fields.
2. **`npm run diagnose:mcp`** shows backend health + proxy status + orphan count in one command.
3. **Watchdog reaper** auto-cleans orphans so they don't accumulate across days.
4. **PreToolUse hook in `~/.claude/settings.json`** blocks Claude Code's native `TaskCreate`/`TaskUpdate`/etc. — ensures even when MCP is flaky, agents can't silently fall back to ephemeral native tracking.

## Longer-term improvements (worth doing)

### a) Make the proxy framing-robust (medium effort)
Modify `mcp-stdio-proxy.js` to either:
- Write BOTH framings simultaneously (the wrong one gets ignored as invalid by most parsers).
- Accept a `--framing=content-length|newline` CLI arg, configured at install time by `configure-mcp.ts` based on detected Claude Code version.

Priority: P2. Addresses possibility #2. Doesn't fix #1 but closes one known vector.

### b) Heartbeat / stall detection on the proxy side (medium effort)
The proxy could track pending requests (by JSON-RPC id). If a response has been sent but no new request arrives for N seconds, log a warning. If pending requests exceed a threshold with no responses from the backend for X seconds, flush a diagnostic to stderr. Doesn't fix the hang, but gives us proof when it recurs.

Priority: P2. Diagnostic aid for future incidents.

### c) Switch to HTTP-native MCP transport (high effort, high impact)
Modern MCP supports direct HTTP transport (no stdio intermediary). Claude Code could connect to `http://127.0.0.1:3456/mcp` directly via the `type: "http"` mcpServer config (as Droid does already — see `scripts/configure-mcp.ts:53-58`).

Eliminates: the proxy process entirely, stdio framing, orphan-proxy bug class, framing auto-detect bug class. Just use plain HTTP like REST.

Blocker: requires Claude Code to support HTTP-transport MCP servers registered via `.mcp.json`. Check current support — if available, migration is ~1 day.

Priority: P1 if supported. Biggest structural fix available.

### d) Client-side retry with REST failover (small effort, defensive)
Wrap MCP tool calls in a helper that times out after N seconds and automatically retries via REST. This is already the *manual* protocol in CLAUDE.md; formalizing it in code would make the fallback automatic.

This requires agent behavior changes, not Priority Forge changes — would live in whatever glue code the agent uses to call the Priority Forge MCP tools.

Priority: P2. Doesn't address root cause but makes it invisible to users.

## What I don't know

- Whether the Claude Code MCP client source exists in `~/Desktop/claude-source-code` — haven't looked since the user said not to touch it. The source may have the client-side code and yield root cause on inspection, but that requires user's explicit permission.
- Exactly which message sequence triggers the wedge. Two observations today, both `complete_task`, not enough to generalize.
- Whether this is new or has been happening for weeks/months. No long-lived logs of MCP activity exist.

## Recommended next steps

In priority order:

1. **Run this session to completion using REST for any further PF ops.** Already our practice; keep it that way until a durable fix.
2. **Test improvement (a) — framing-robust proxy.** Low risk, cheap, closes one known failure mode. Worth doing regardless of what else is true.
3. **Investigate (c) — HTTP-transport MCP.** If Claude Code supports `type: "http"` in `.mcp.json`, migration eliminates the entire stdio layer and the whole class of bugs we've been fighting.
4. **If hang recurs AND we want root-cause proof:** get explicit permission to read Claude Code's MCP client source in `~/Desktop/claude-source-code`, find the request-tracking code, identify the state machine.

Until one of those ships, the hang is a rare-but-real tail event. REST fallback handles it without data loss.

---

## Resolution — 2026-04-13 (HTTP-transport migration)

Step 3 from "Recommended next steps" above shipped the same day this RCA was written. Claude Code does support HTTP-transport MCP servers; we'd just never used the feature for it (Cursor and Droid have used it all along — see `scripts/configure-mcp.ts` lines 49–71 for the historical config divergence).

### What changed

1. **All five `.mcp.json` discovery files** rewritten from stdio-proxy to HTTP transport:
   - `~/.mcp.json`
   - `~/Desktop/.mcp.json`
   - `~/Desktop/github/.mcp.json`
   - `~/Desktop/github/priority-forge/.mcp.json`
   - `~/Desktop/github/postit/.mcp.json` *(new — created during migration; was missing before)*

   Each now contains:
   ```json
   {
     "mcpServers": {
       "priority-forge": {
         "type": "http",
         "url": "http://127.0.0.1:3456/mcp"
       }
     }
   }
   ```

2. **`scripts/configure-mcp.ts` patched** so future `npm run setup:mcp` runs default to HTTP. The stdio proxy installer is preserved but only fires when `PRIORITY_FORGE_USE_STDIO=1` is set, as a documented escape hatch.

3. **Backend already supported it.** The Express app exposes `POST /mcp` speaking streamable-HTTP MCP (protocol version `2025-03-26`). Smoke test:
   ```bash
   curl -X POST http://127.0.0.1:3456/mcp \
     -H "Content-Type: application/json" \
     -H "Accept: application/json, text/event-stream" \
     -d '{"jsonrpc":"2.0","id":1,"method":"initialize",...}'
   # → 200, full handshake, 34 tools listed via tools/list
   ```

### Why this fixes the wedge by construction

The original RCA localized the bug to one of two places, both inside Claude Code's process:
- (a) proxy-written bytes the client wasn't reading, OR
- (b) bytes read but lost in the client's stdio state machine.

HTTP transport eliminates *both* possible loci because there are no bytes flowing through stdio at all. Each tool call is a normal HTTP request/response handled by Claude Code's HTTP client (the same one that handles every other network call) instead of by the bespoke MCP-over-stdio framing/state-machine code. The wedge class doesn't have a place to live.

### What stays

- **REST fallback in `~/.claude/CLAUDE.md`** — still mandatory protocol if MCP ever fails, regardless of transport. No change.
- **Stdio proxy code at `scripts/mcp-stdio-proxy.js`** — still installed to `~/.local/share/priority-forge/mcp-proxy.js` on every `npm run setup:mcp` run. Activated by `PRIORITY_FORGE_USE_STDIO=1`. Kept in case a future Claude Code release breaks HTTP transport.
- **Watchdog reaper, signal handlers, framing auto-detect** — orthogonal to the wedge class but still useful for the stdio fallback path.

### Open issues surfaced during migration

- **Programmatic trust pre-acceptance is not a reliable installation path.** Live testing showed Claude Code can use a project-scoped `.mcp.json` server while `~/.claude.json` still has empty `enabledMcpjsonServers` and `hasTrustDialogAccepted: false`. We therefore removed the unsupported `acceptMcpTrust()` write path from `configure-mcp.ts` and treat first-use approval for project-scoped servers as expected behavior. Users who want a personal no-prompt setup should add a local-scoped server with `claude mcp add --scope local ...`.

### Validation

- Backend HTTP MCP endpoint returns full tool surface (34 tools) over `text/event-stream`.
- `configure-mcp.ts` typechecks clean (after fixing two pre-existing tsc warnings surfaced during this work — see PF tasks closed 2026-04-13).
- Per-session validation requires a Claude Code restart; the current session is bonded to its launch-time stdio proxy.

### What the wedge class looks like in production after this

Should be: zero recurrences, since the offending code path is no longer reachable for `priority-forge`. If a hang ever recurs, it would have to be a new bug class (HTTP timeout, backend stall, etc.) and should be investigated as such — do not assume it's the same wedge.
