paperclip/server/src/services
Devin Foley 911a1e8b0d
Fix continuation recovery retry streaks by failure cause (#7031)
## Thinking Path

> - Paperclip orchestrates AI agents for zero-human companies.
> - The recovery subsystem is responsible for keeping assigned work
moving when a live heartbeat run disappears or fails.
> - `continuation_recovery` is the path that re-enqueues stranded
`in_progress` issues after an interrupted continuation attempt.
> - That path recently gained cause-aware retry classes and transient
retry caps, but the streak counter was still aggregating mixed failure
causes into one retry history.
> - That meant a sequence like `timeout -> timeout -> adapter_failed ->
adapter_failed` could escalate as a false `3x adapter_failed` streak
even though the latest cause had only happened twice.
> - This pull request makes continuation retry streaks count only
consecutive failures whose `errorCode` matches the latest run and adds a
regression test for the mixed-cause case.
> - The benefit is that transient retry backoff and escalation now match
the actual current failure cause instead of inheriting stale budget from
unrelated failures.

## What Changed

- Updated `summarizeRecentContinuationRetries(...)` to stop counting as
soon as the continuation failure cause no longer matches the latest
run's `errorCode`.
- Wired the continuation recovery escalation/backoff path to pass the
latest classified `errorCode` into the retry streak summarizer.
- Added a regression test proving mixed-cause continuation failures do
not consume the transient retry cap for a new failure cause.

## Verification

- `pnpm exec vitest run
server/src/__tests__/heartbeat-process-recovery.test.ts`

## Risks

- Low risk. The behavioral change is intentionally narrow, but any
future continuation retry modes that rely on `errorCode = null` will now
be counted as a separate streak bucket and should be kept in mind when
adding new retry classifications.

## Model Used

- OpenAI Codex via Paperclip `codex_local` (GPT-5-based Codex coding
agent; exact backend revision is not surfaced in the runtime), with tool
use, shell execution, and patch application in the local repository.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] If this change affects the UI, I have included before/after
screenshots
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-05-29 19:48:59 -07:00
..
recovery Fix continuation recovery retry streaks by failure cause (#7031) 2026-05-29 19:48:59 -07:00
access.ts [codex] Add agent permissions and controls plan (#6386) 2026-05-22 08:12:52 -05:00
activity-log.ts [codex] Add plugin orchestration host APIs (#4114) 2026-04-20 08:52:51 -05:00
activity.ts Add sandbox environment support (#4415) 2026-04-24 12:15:53 -07:00
adapter-plugin-store.ts [codex] Add runtime lifecycle recovery and live issue visibility (#4419) 2026-04-24 15:50:32 -05:00
agent-instructions.ts chore: mark bootstrapPromptTemplate as deprecated 2026-03-26 11:12:25 -05:00
agent-permissions.ts [codex] Add agent permissions and controls plan (#6386) 2026-05-22 08:12:52 -05:00
agent-start-lock.ts [codex] Add runtime lifecycle recovery and live issue visibility (#4419) 2026-04-24 15:50:32 -05:00
agents.ts [codex] Add agent permissions and controls plan (#6386) 2026-05-22 08:12:52 -05:00
approvals.ts Add username log censor setting 2026-03-20 08:50:00 -05:00
assets.ts refactor: rename packages to @paperclipai and CLI binary to paperclipai 2026-03-03 08:45:26 -06:00
authorization.ts [codex] Add agent permissions and controls plan (#6386) 2026-05-22 08:12:52 -05:00
board-auth.ts feat: implement multi-user access and invite flows (#3784) 2026-04-17 09:44:19 -05:00
budgets.ts Sync/master post pap1497 followups 2026 04 15 (#3779) 2026-04-15 21:13:56 -05:00
catalog-provenance.ts [codex] Add skills CLI and catalog management (#6782) 2026-05-28 07:33:51 -10:00
cloud-upstreams.ts [codex] Bundle local branch fixes from PAP-10032 (#6604) 2026-05-25 07:25:26 -05:00
companies.ts Fix wrapped company issue prefix conflicts (#6423) 2026-05-22 15:27:54 -05:00
company-export-readme.ts fix: link Agent Company to agentcompanies.io in export README 2026-03-20 08:06:04 -05:00
company-member-roles.ts [codex] Add agent permissions and controls plan (#6386) 2026-05-22 08:12:52 -05:00
company-portability.ts [codex] Add skills CLI and catalog management (#6782) 2026-05-28 07:33:51 -10:00
company-search-rate-limit.ts Add full company search page (#5293) 2026-05-06 06:32:37 -05:00
company-search.ts Add full company search page (#5293) 2026-05-06 06:32:37 -05:00
company-skills.ts [codex] Add skills CLI and catalog management (#6782) 2026-05-28 07:33:51 -10:00
costs.ts Improve operator workflow QoL (#5291) 2026-05-06 06:30:44 -05:00
cron.ts Add plugin framework and settings UI 2026-03-13 16:22:34 -05:00
dashboard.ts [codex] Harden heartbeat scheduling and runtime controls (#4223) 2026-04-21 12:24:11 -05:00
default-agent-instructions.ts Add default agent instructions bundle 2026-03-20 07:42:36 -05:00
document-annotations.ts [codex] Add document annotations and comments (#6733) 2026-05-26 06:41:23 -07:00
documents.ts [codex] Add issue document locking (#6009) 2026-05-15 08:54:55 -05:00
environment-config.ts Add secrets provider vaults and remote import (#5429) 2026-05-09 18:22:17 -05:00
environment-execution-target.ts Fix exe.dev sandbox installs for gemini/opencode local adapters (#5737) 2026-05-11 14:28:22 -07:00
environment-probe.ts Add sandbox environment support (#4415) 2026-04-24 12:15:53 -07:00
environment-run-orchestrator.ts Add sandbox callback bridge for remote environment API access (#4801) 2026-04-29 16:37:34 -07:00
environment-runtime.ts Add Cloudflare sandbox provider plugin (#5687) 2026-05-11 07:33:13 -07:00
environments.ts Add sandbox environment support (#4415) 2026-04-24 12:15:53 -07:00
execution-workspace-policy.ts PAPA-430: workspace finalize gates + no-remote-git enforcement (#6969) 2026-05-29 08:25:29 -07:00
execution-workspaces.ts [codex] Bundle local branch fixes from PAP-10032 (#6604) 2026-05-25 07:25:26 -05:00
feedback-redaction.ts Add feedback voting and thumbs capture flow 2026-04-02 09:11:49 -05:00
feedback-share-client.ts Restore feedback trace export fixes 2026-04-03 15:59:42 -05:00
feedback.ts Add recovery handoff system notices (#5289) 2026-05-06 06:05:58 -05:00
finance.ts Sync/master post pap1497 followups 2026 04 15 (#3779) 2026-04-15 21:13:56 -05:00
github-fetch.ts fix: harden GHE URL detection and extract shared GitHub helpers 2026-04-01 21:05:48 +00:00
goals.ts Improve onboarding defaults and issue goal fallback 2026-03-12 08:50:31 -05:00
heartbeat-run-summary.ts [codex] Add run liveness continuations (#4083) 2026-04-20 06:01:49 -05:00
heartbeat-stop-metadata.test.ts [codex] Retry max-turn exhausted heartbeats (#5096) 2026-05-03 11:30:48 -05:00
heartbeat-stop-metadata.ts [codex] Retry max-turn exhausted heartbeats (#5096) 2026-05-03 11:30:48 -05:00
heartbeat.ts PAPA-430: workspace finalize gates + no-remote-git enforcement (#6969) 2026-05-29 08:25:29 -07:00
hire-hook.ts fix(adapters): honor paused overrides and isolate UI parser state 2026-04-04 14:04:33 -05:00
inbox-dismissals.ts Persist non-issue inbox dismissals 2026-04-09 06:16:05 -05:00
index.ts [codex] Add document annotations and comments (#6733) 2026-05-26 06:41:23 -07:00
instance-settings.ts Add accepted-plan decomposition exact-once guards and UI state (#6831) 2026-05-28 23:30:18 -07:00
invite-grants.ts feat: implement multi-user access and invite flows (#3784) 2026-04-17 09:44:19 -05:00
issue-approvals.ts refactor: rename packages to @paperclipai and CLI binary to paperclipai 2026-03-03 08:45:26 -06:00
issue-assignment-wakeup.ts fix: close remaining routine merge blockers 2026-03-20 16:40:27 -05:00
issue-continuation-summary.ts fix: harden release registry verification against npm lag (#4816) 2026-05-09 22:18:12 -07:00
issue-execution-policy.ts [codex] Add issue monitor liveness controls (#4988) 2026-05-03 08:58:53 -05:00
issue-goal-fallback.ts Seed onboarding project and issue goal context 2026-03-24 11:48:59 -05:00
issue-liveness.ts [codex] Add runtime lifecycle recovery and live issue visibility (#4419) 2026-04-24 15:50:32 -05:00
issue-recovery-actions.ts [codex] Add source-scoped recovery actions (#5599) 2026-05-12 09:37:15 -05:00
issue-references.ts [codex] Add document annotations and comments (#6733) 2026-05-26 06:41:23 -07:00
issue-thread-interactions.test.ts [codex] Add structured issue-thread interactions (#4244) 2026-04-21 20:15:11 -05:00
issue-thread-interactions.ts PAPA-430: workspace finalize gates + no-remote-git enforcement (#6969) 2026-05-29 08:25:29 -07:00
issue-tree-control.ts [codex] Split backend control-plane QoL slice (#4700) 2026-04-28 16:46:45 -05:00
issues.ts PAPA-430: workspace finalize gates + no-remote-git enforcement (#6969) 2026-05-29 08:25:29 -07:00
json-schema-secret-refs.ts Generalize sandbox provider core for plugin-only providers (#4449) 2026-04-24 18:03:41 -07:00
live-events.ts Add plugin framework and settings UI 2026-03-13 16:22:34 -05:00
local-service-supervisor.ts fix: harden heartbeat and adapter runtime workflows 2026-04-10 22:26:21 -05:00
plugin-capability-validator.ts [codex] Add agent permissions and controls plan (#6386) 2026-05-22 08:12:52 -05:00
plugin-config-validator.ts Refactor secret-ref format registration to use a UI hint for Paperclip secret UUIDs 2026-03-14 15:43:56 -07:00
plugin-database.ts Fix LLM Wiki package and migration validation (#6010) 2026-05-15 10:20:02 -05:00
plugin-dev-watcher.ts [codex] Improve local plugin development workflow (#5821) 2026-05-12 17:38:24 -05:00
plugin-environment-driver.ts fix(plugin): raise environmentProbe RPC timeout to 120s for cold-start sandboxes (#6289) 2026-05-18 09:32:12 -07:00
plugin-event-bus.ts Simplify plugin runtime and cleanup lifecycle 2026-03-13 16:58:29 -05:00
plugin-host-service-cleanup.ts Add plugin framework and settings UI 2026-03-13 16:22:34 -05:00
plugin-host-services.ts [codex] Add agent permissions and controls plan (#6386) 2026-05-22 08:12:52 -05:00
plugin-job-coordinator.ts Add plugin framework and settings UI 2026-03-13 16:22:34 -05:00
plugin-job-scheduler.ts Add plugin framework and settings UI 2026-03-13 16:22:34 -05:00
plugin-job-store.ts Add plugin framework and settings UI 2026-03-13 16:22:34 -05:00
plugin-lifecycle.ts [codex] Runtime control-plane fixes (#6380) 2026-05-20 10:37:11 -05:00
plugin-loader.ts fix(remote-sandbox): harden host workspace resumes (#5922) 2026-05-13 16:23:04 -05:00
plugin-local-folders.ts [codex] Roll up May 17 branch changes (#6210) 2026-05-17 17:15:06 -05:00
plugin-log-retention.ts Add plugin framework and settings UI 2026-03-13 16:22:34 -05:00
plugin-managed-agents.ts [codex] Add LLM Wiki plugin host support (#5597) 2026-05-10 07:34:12 -05:00
plugin-managed-routines.ts Expand plugin host surface (#5205) 2026-05-05 07:42:57 -05:00
plugin-managed-skills.ts [codex] Add LLM Wiki plugin host support (#5597) 2026-05-10 07:34:12 -05:00
plugin-manifest-validator.ts Add plugin framework and settings UI 2026-03-13 16:22:34 -05:00
plugin-registry.ts Expand plugin host surface (#5205) 2026-05-05 07:42:57 -05:00
plugin-runtime-sandbox.ts Add plugin framework and settings UI 2026-03-13 16:22:34 -05:00
plugin-secrets-handler.ts Add secrets provider vaults and remote import (#5429) 2026-05-09 18:22:17 -05:00
plugin-state-store.ts Add plugin framework and settings UI 2026-03-13 16:22:34 -05:00
plugin-stream-bus.ts Add plugin framework and settings UI 2026-03-13 16:22:34 -05:00
plugin-tool-dispatcher.ts Add plugin framework and settings UI 2026-03-13 16:22:34 -05:00
plugin-tool-registry.ts Add plugin framework and settings UI 2026-03-13 16:22:34 -05:00
plugin-worker-manager.ts [codex] Bundle local branch fixes from PAP-10032 (#6604) 2026-05-25 07:25:26 -05:00
portable-path.ts [codex] Add skills CLI and catalog management (#6782) 2026-05-28 07:33:51 -10:00
principal-access-compatibility.ts [codex] Add agent permissions and controls plan (#6386) 2026-05-22 08:12:52 -05:00
productivity-review.ts Guard cheap recovery model usage (#6371) 2026-05-19 13:46:02 -05:00
project-workspace-runtime-config.ts [codex] Respect manual workspace runtime controls (#4125) 2026-04-20 10:39:37 -05:00
projects.ts Expand plugin host surface (#5205) 2026-05-05 07:42:57 -05:00
quota-windows.ts feat(costs): add billing, quota, and budget control plane 2026-03-16 15:11:01 -05:00
resource-memberships.ts [codex] Add resource membership controls (#6677) 2026-05-25 13:12:41 -05:00
routines.ts [codex] Add routine env secrets support (#6212) 2026-05-17 16:30:34 -05:00
run-continuations.ts [codex] Add runtime lifecycle recovery and live issue visibility (#4419) 2026-04-24 15:50:32 -05:00
run-liveness.ts [codex] Add runtime lifecycle recovery and live issue visibility (#4419) 2026-04-24 15:50:32 -05:00
run-log-store.ts [codex] Add runtime lifecycle recovery and live issue visibility (#4419) 2026-04-24 15:50:32 -05:00
sandbox-provider-runtime.ts Generalize sandbox provider core for plugin-only providers (#4449) 2026-04-24 18:03:41 -07:00
secrets.ts [codex] Provider vault secrets UX (#6381) 2026-05-19 15:50:23 -05:00
session-workspace-cwd.test.ts fix(remote-sandbox): harden host workspace resumes (#5922) 2026-05-13 16:23:04 -05:00
session-workspace-cwd.ts fix(remote-sandbox): harden host workspace resumes (#5922) 2026-05-13 16:23:04 -05:00
sidebar-badges.ts Persist non-issue inbox dismissals 2026-04-09 06:16:05 -05:00
sidebar-preferences.ts [codex] Improve workspace runtime and navigation ergonomics (#3680) 2026-04-14 12:57:11 -05:00
skills-catalog.ts [codex] Add skills CLI and catalog management (#6782) 2026-05-28 07:33:51 -10:00
work-products.ts Address remaining Greptile workspace review 2026-03-17 10:12:44 -05:00
workspace-operation-log-store.ts Add workspace operation tracking and fix project properties JSX 2026-03-17 09:36:35 -05:00
workspace-operations.ts [codex] Improve agent runtime recovery and governance (#4086) 2026-04-20 06:19:48 -05:00
workspace-realization.ts Add sandbox environment support (#4415) 2026-04-24 12:15:53 -07:00
workspace-runtime-read-model.ts Fix workspace runtime state reconciliation 2026-04-04 17:48:54 -05:00
workspace-runtime.ts [codex] Add skills CLI and catalog management (#6782) 2026-05-28 07:33:51 -10:00