Open

Heartbeat silently dies after restart: startup recovery marks the pending row 'missed', then supersede CAS fails silently — no run ever starts

BUG: the heartbeat service can go permanently silent after a restart, and from the user's side it is indistinguishable from the assistant simply having nothing to say. Repro (observed Sep 30 – Oct 5 2026, 6 days with zero heartbeat runs): 1. Every heartbeat row is inserted as status='pending' with scheduled_for = now + intervalMs (verified: all superseded rows sit at exactly +60.0 min from created_at). 2. On service start, start() runs startup recovery, which calls markStaleRunsAsMissed(threshold = 5 min). The orphaned pending row from before the restart has scheduled_for up to 60 min in the future and created_at in the past, so it flips to 'missed'. 3. startIntervalMode() -> scheduleNextRun() then calls supersedePendingRun(this._pendingRunId). That is a CAS matching only status='pending', so it FAILS SILENTLY against the now-'missed' row. _pendingRunId is left pointing at a dead row. 4. scheduleNextRun() writes a brand-new pending row for now + intervalMs. The next cycle repeats. Net effect: rows accumulate, the previous one is superseded or aged out, and no row ever transitions to 'running'. Superseded rows have started_at = NULL. Evidence: superseded rows appear at ~142-180 min intervals (restart boundaries), not the 60-min timer cadence. Related smell: skip call sites guard with 'if (!this.cronMode) this.scheduleNextRun(...)' but several supersede call sites (heartbeat-service.ts ~253, 424, 449, 483, 690) do not, so lifecycle paths can supersede and re-arm inconsistently. Suggested fix: reconcile the orphaned pending row during startup instead of superseding a row that recovery already marked 'missed' — e.g. clear _pendingRunId when the stale-marking pass touches it, or run markStaleRunsAsMissed after pending-row reconciliation. Impact: 207-hour gap between consecutive heartbeat conversations (Sep 27 23:38 -> Oct 6 14:48). Promised follow-ups made inside a heartbeat never get carried forward, and the run ledger partially conceals the gap (4 'ok' rows vs 5 actual heartbeat conversations). It self-healed on Oct 6 13:48 only because a run won a race against the 5-minute stale threshold — it will recur. Files: src/heartbeat/heartbeat-service.ts, src/heartbeat/heartbeat-run-store.ts

0 commentsby Cade

Comments (0)

Loading…

The Personal AI you were promised

Get Started →