[codex] Add run liveness continuations (#4083)
## Thinking Path
> - Paperclip orchestrates AI agents for zero-human companies.
> - Heartbeat runs are the control-plane record of each agent execution
window.
> - Long-running local agents can exhaust context or stop while still
holding useful next-step state.
> - Operators need that stop reason, next action, and continuation path
to be durable and visible.
> - This pull request adds run liveness metadata, continuation
summaries, and UI surfaces for issue run ledgers.
> - The benefit is that interrupted or long-running work can resume with
clearer context instead of losing the agent's last useful handoff.
## What Changed
- Added heartbeat-run liveness fields, continuation attempt tracking,
and an idempotent `0058` migration.
- Added server services and tests for run liveness, continuation
summaries, stop metadata, and activity backfill.
- Wired local and HTTP adapters to surface continuation/liveness context
through shared adapter utilities.
- Added shared constants, validators, and heartbeat types for liveness
continuation state.
- Added issue-detail UI surfaces for continuation handoffs and the run
ledger, with component tests.
- Updated agent runtime docs, heartbeat protocol docs, prompt guidance,
onboarding assets, and skills instructions to explain continuation
behavior.
- Addressed Greptile feedback by scoping document evidence by run,
excluding system continuation-summary documents from liveness evidence,
importing shared liveness types, surfacing hidden ledger run counts,
documenting bounded retry behavior, and moving run-ledger liveness
backfill off the request path.
## Verification
- `pnpm exec vitest run packages/adapter-utils/src/server-utils.test.ts
server/src/__tests__/run-continuations.test.ts
server/src/__tests__/run-liveness.test.ts
server/src/__tests__/activity-service.test.ts
server/src/__tests__/documents-service.test.ts
server/src/__tests__/issue-continuation-summary.test.ts
server/src/services/heartbeat-stop-metadata.test.ts
ui/src/components/IssueRunLedger.test.tsx
ui/src/components/IssueContinuationHandoff.test.tsx
ui/src/components/IssueDocumentsSection.test.tsx`
- `pnpm --filter @paperclipai/db build`
- `pnpm exec vitest run server/src/__tests__/activity-service.test.ts
ui/src/components/IssueRunLedger.test.tsx`
- `pnpm --filter @paperclipai/ui typecheck`
- `pnpm --filter @paperclipai/server typecheck`
- `pnpm exec vitest run server/src/__tests__/activity-service.test.ts
server/src/__tests__/run-continuations.test.ts
ui/src/components/IssueRunLedger.test.tsx`
- `pnpm exec vitest run
server/src/__tests__/heartbeat-process-recovery.test.ts -t "treats a
plan document update"`
- `pnpm exec vitest run server/src/__tests__/activity-service.test.ts
server/src/__tests__/heartbeat-process-recovery.test.ts -t "activity
service|treats a plan document update"`
- Remote PR checks on head `e53b1a1d`: `verify`, `e2e`, `policy`, and
Snyk all passed.
- Confirmed `public-gh/master` is an ancestor of this branch after
fetching `public-gh master`.
- Confirmed `pnpm-lock.yaml` is not included in the branch diff.
- Confirmed migration `0058_wealthy_starbolt.sql` is ordered after
`0057` and uses `IF NOT EXISTS` guards for repeat application.
- Greptile inline review threads are resolved.
## Risks
- Medium risk: this touches heartbeat execution, liveness recovery,
activity rendering, issue routes, shared contracts, docs, and UI.
- Migration risk is mitigated by additive columns/indexes and idempotent
guards.
- Run-ledger liveness backfill is now asynchronous, so the first ledger
response can briefly show historical missing liveness until the
background backfill completes.
- UI screenshot coverage is not included in this packaging pass;
validation is currently through focused component tests.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- OpenAI Codex, GPT-5.4, local tool-use coding agent with terminal, git,
GitHub connector, GitHub CLI, and Paperclip API access.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] If this change affects the UI, I have included before/after
screenshots
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] I will address all Greptile and reviewer comments before
requesting merge
Screenshot note: no before/after screenshots were captured in this PR
packaging pass; the UI changes are covered by focused component tests
listed above.
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-04-20 06:01:49 -05:00
|
|
|
import { describe, expect, it } from "vitest";
|
|
|
|
|
import {
|
|
|
|
|
buildHeartbeatRunStopMetadata,
|
|
|
|
|
mergeHeartbeatRunStopMetadata,
|
|
|
|
|
resolveHeartbeatRunTimeoutPolicy,
|
|
|
|
|
} from "./heartbeat-stop-metadata.js";
|
|
|
|
|
|
|
|
|
|
describe("heartbeat stop metadata", () => {
|
|
|
|
|
it("keeps local coding adapters at no timeout by default", () => {
|
|
|
|
|
for (const adapterType of [
|
|
|
|
|
"codex_local",
|
|
|
|
|
"claude_local",
|
|
|
|
|
"cursor",
|
|
|
|
|
"gemini_local",
|
|
|
|
|
"opencode_local",
|
|
|
|
|
"pi_local",
|
|
|
|
|
"process",
|
|
|
|
|
]) {
|
|
|
|
|
expect(resolveHeartbeatRunTimeoutPolicy(adapterType, {})).toEqual({
|
|
|
|
|
effectiveTimeoutSec: 0,
|
|
|
|
|
timeoutConfigured: false,
|
|
|
|
|
timeoutSource: "default",
|
|
|
|
|
});
|
|
|
|
|
}
|
|
|
|
|
});
|
|
|
|
|
|
|
|
|
|
it("records configured timeout policy and timeout stop reason", () => {
|
|
|
|
|
const metadata = buildHeartbeatRunStopMetadata({
|
|
|
|
|
adapterType: "codex_local",
|
|
|
|
|
adapterConfig: { timeoutSec: 45 },
|
|
|
|
|
outcome: "timed_out",
|
|
|
|
|
errorCode: "timeout",
|
|
|
|
|
errorMessage: "Timed out after 45s",
|
|
|
|
|
});
|
|
|
|
|
|
|
|
|
|
expect(metadata).toEqual({
|
|
|
|
|
effectiveTimeoutSec: 45,
|
|
|
|
|
timeoutConfigured: true,
|
|
|
|
|
timeoutSource: "config",
|
|
|
|
|
stopReason: "timeout",
|
|
|
|
|
timeoutFired: true,
|
|
|
|
|
});
|
|
|
|
|
});
|
|
|
|
|
|
|
|
|
|
it("distinguishes budget cancellation from manual cancellation", () => {
|
|
|
|
|
expect(
|
|
|
|
|
buildHeartbeatRunStopMetadata({
|
|
|
|
|
adapterType: "codex_local",
|
|
|
|
|
adapterConfig: {},
|
|
|
|
|
outcome: "cancelled",
|
|
|
|
|
errorCode: "cancelled",
|
|
|
|
|
errorMessage: "Cancelled due to budget pause",
|
|
|
|
|
}).stopReason,
|
|
|
|
|
).toBe("budget_paused");
|
|
|
|
|
|
|
|
|
|
expect(
|
|
|
|
|
buildHeartbeatRunStopMetadata({
|
|
|
|
|
adapterType: "codex_local",
|
|
|
|
|
adapterConfig: {},
|
|
|
|
|
outcome: "cancelled",
|
|
|
|
|
errorCode: "cancelled",
|
|
|
|
|
errorMessage: "Cancelled by control plane",
|
|
|
|
|
}).stopReason,
|
|
|
|
|
).toBe("cancelled");
|
|
|
|
|
});
|
|
|
|
|
|
[codex] Retry max-turn exhausted heartbeats (#5096)
## Thinking Path
> - Paperclip orchestrates AI agents for autonomous companies, and
heartbeat execution is the control-plane loop that keeps assigned work
moving.
> - Max-turn exhaustion is a recoverable local-adapter stop condition
for Claude and Gemini agents when a run needs another heartbeat to
continue safely.
> - The previous behavior could leave max-turn continuation details hard
to inspect, and duplicate/stale continuation wakes could keep running
after issue state changed.
> - The adapter layer also needed to avoid trusting arbitrary
stdout/stderr text as scheduler control metadata.
> - This pull request adds bounded max-turn continuation scheduling,
visible retry state, structured stop metadata handling, and
stale/duplicate continuation guards.
> - The benefit is safer automatic continuation after max-turn stops,
clearer operator visibility, and fewer duplicate or stale agent runs.
## What Changed
- Replaces closed PR #4952, whose head repository was deleted.
- Rebases the recovered max-turn continuation branch onto current
`paperclipai/paperclip:master`.
- Adds max-turn continuation scheduling and retry-state plumbing for
heartbeat runs.
- Adds stale/duplicate continuation suppression when issue status,
ownership, or execution locks change.
- Normalizes Claude/Gemini max-turn detection around structured stop
metadata instead of unstructured stdout/stderr text.
- Surfaces max-turn continuation settings and retry visibility in the
board UI.
- Adds focused server, adapter, and UI tests for max-turn stop metadata,
retry scheduling, stale queued-run invalidation, adapter
parsing/execution, run ledger display, and agent config patching.
## Verification
- `pnpm install --no-frozen-lockfile` to refresh local dependencies
after rebasing onto current `master`.
- `pnpm run preflight:workspace-links && pnpm exec vitest run
server/src/__tests__/claude-local-adapter.test.ts
server/src/__tests__/claude-local-execute.test.ts
server/src/__tests__/gemini-local-adapter.test.ts
server/src/__tests__/gemini-local-execute.test.ts
server/src/__tests__/heartbeat-retry-scheduling.test.ts
server/src/__tests__/heartbeat-stale-queue-invalidation.test.ts
server/src/services/heartbeat-stop-metadata.test.ts
ui/src/components/IssueRunLedger.test.tsx
ui/src/lib/agent-config-patch.test.ts ui/src/lib/runRetryState.test.ts
--testTimeout=20000`
- `pnpm --filter @paperclipai/adapter-claude-local typecheck && pnpm
--filter @paperclipai/adapter-gemini-local typecheck && pnpm --filter
@paperclipai/server typecheck && pnpm --filter @paperclipai/ui
typecheck`
- UI screenshot note: the UI changes are limited to config/ledger state
rendering rather than layout changes; component/unit coverage above
verifies the rendered behavior.
## Risks
- Medium behavior risk: heartbeat retry gating now suppresses max-turn
continuations when issue state or execution locks drift, so any callers
that relied on stale continuations running will now see cancellation
instead.
- Low adapter risk: Claude/Gemini unstructured text no longer triggers
max-turn scheduler metadata, so only structured stop signals and Gemini
exit code 53 are trusted.
- No database migrations.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- OpenAI Codex coding agent, GPT-5-class model, tool-enabled local
repository editing and command execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] If this change affects the UI, I have included before/after
screenshots (not applicable: state/default rendering only; covered by
component/unit tests)
- [x] I have updated relevant documentation to reflect my changes (not
applicable: no user-facing command or docs contract changed)
- [x] I have considered and documented any risks above
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-05-03 11:30:48 -05:00
|
|
|
it("normalizes max-turn exhaustion stop reasons", () => {
|
|
|
|
|
expect(
|
|
|
|
|
buildHeartbeatRunStopMetadata({
|
|
|
|
|
adapterType: "claude_local",
|
|
|
|
|
adapterConfig: {},
|
|
|
|
|
outcome: "failed",
|
|
|
|
|
errorCode: "turn_limit_exhausted",
|
|
|
|
|
errorMessage: "turn limit reached",
|
|
|
|
|
}).stopReason,
|
|
|
|
|
).toBe("max_turns_exhausted");
|
|
|
|
|
|
|
|
|
|
const merged = mergeHeartbeatRunStopMetadata(
|
|
|
|
|
{ stopReason: "turn_limit_exhausted" },
|
|
|
|
|
buildHeartbeatRunStopMetadata({
|
|
|
|
|
adapterType: "claude_local",
|
|
|
|
|
adapterConfig: {},
|
|
|
|
|
outcome: "failed",
|
|
|
|
|
errorCode: "adapter_failed",
|
|
|
|
|
}),
|
|
|
|
|
);
|
|
|
|
|
expect(merged.stopReason).toBe("max_turns_exhausted");
|
|
|
|
|
});
|
|
|
|
|
|
|
|
|
|
it("prioritizes succeeded outcome over inconsistent max-turn error metadata", () => {
|
|
|
|
|
expect(
|
|
|
|
|
buildHeartbeatRunStopMetadata({
|
|
|
|
|
adapterType: "claude_local",
|
|
|
|
|
adapterConfig: {},
|
|
|
|
|
outcome: "succeeded",
|
|
|
|
|
errorCode: "max_turns_exhausted",
|
|
|
|
|
}).stopReason,
|
|
|
|
|
).toBe("completed");
|
|
|
|
|
});
|
|
|
|
|
|
[codex] Add run liveness continuations (#4083)
## Thinking Path
> - Paperclip orchestrates AI agents for zero-human companies.
> - Heartbeat runs are the control-plane record of each agent execution
window.
> - Long-running local agents can exhaust context or stop while still
holding useful next-step state.
> - Operators need that stop reason, next action, and continuation path
to be durable and visible.
> - This pull request adds run liveness metadata, continuation
summaries, and UI surfaces for issue run ledgers.
> - The benefit is that interrupted or long-running work can resume with
clearer context instead of losing the agent's last useful handoff.
## What Changed
- Added heartbeat-run liveness fields, continuation attempt tracking,
and an idempotent `0058` migration.
- Added server services and tests for run liveness, continuation
summaries, stop metadata, and activity backfill.
- Wired local and HTTP adapters to surface continuation/liveness context
through shared adapter utilities.
- Added shared constants, validators, and heartbeat types for liveness
continuation state.
- Added issue-detail UI surfaces for continuation handoffs and the run
ledger, with component tests.
- Updated agent runtime docs, heartbeat protocol docs, prompt guidance,
onboarding assets, and skills instructions to explain continuation
behavior.
- Addressed Greptile feedback by scoping document evidence by run,
excluding system continuation-summary documents from liveness evidence,
importing shared liveness types, surfacing hidden ledger run counts,
documenting bounded retry behavior, and moving run-ledger liveness
backfill off the request path.
## Verification
- `pnpm exec vitest run packages/adapter-utils/src/server-utils.test.ts
server/src/__tests__/run-continuations.test.ts
server/src/__tests__/run-liveness.test.ts
server/src/__tests__/activity-service.test.ts
server/src/__tests__/documents-service.test.ts
server/src/__tests__/issue-continuation-summary.test.ts
server/src/services/heartbeat-stop-metadata.test.ts
ui/src/components/IssueRunLedger.test.tsx
ui/src/components/IssueContinuationHandoff.test.tsx
ui/src/components/IssueDocumentsSection.test.tsx`
- `pnpm --filter @paperclipai/db build`
- `pnpm exec vitest run server/src/__tests__/activity-service.test.ts
ui/src/components/IssueRunLedger.test.tsx`
- `pnpm --filter @paperclipai/ui typecheck`
- `pnpm --filter @paperclipai/server typecheck`
- `pnpm exec vitest run server/src/__tests__/activity-service.test.ts
server/src/__tests__/run-continuations.test.ts
ui/src/components/IssueRunLedger.test.tsx`
- `pnpm exec vitest run
server/src/__tests__/heartbeat-process-recovery.test.ts -t "treats a
plan document update"`
- `pnpm exec vitest run server/src/__tests__/activity-service.test.ts
server/src/__tests__/heartbeat-process-recovery.test.ts -t "activity
service|treats a plan document update"`
- Remote PR checks on head `e53b1a1d`: `verify`, `e2e`, `policy`, and
Snyk all passed.
- Confirmed `public-gh/master` is an ancestor of this branch after
fetching `public-gh master`.
- Confirmed `pnpm-lock.yaml` is not included in the branch diff.
- Confirmed migration `0058_wealthy_starbolt.sql` is ordered after
`0057` and uses `IF NOT EXISTS` guards for repeat application.
- Greptile inline review threads are resolved.
## Risks
- Medium risk: this touches heartbeat execution, liveness recovery,
activity rendering, issue routes, shared contracts, docs, and UI.
- Migration risk is mitigated by additive columns/indexes and idempotent
guards.
- Run-ledger liveness backfill is now asynchronous, so the first ledger
response can briefly show historical missing liveness until the
background backfill completes.
- UI screenshot coverage is not included in this packaging pass;
validation is currently through focused component tests.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- OpenAI Codex, GPT-5.4, local tool-use coding agent with terminal, git,
GitHub connector, GitHub CLI, and Paperclip API access.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] If this change affects the UI, I have included before/after
screenshots
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] I will address all Greptile and reviewer comments before
requesting merge
Screenshot note: no before/after screenshots were captured in this PR
packaging pass; the UI changes are covered by focused component tests
listed above.
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-04-20 06:01:49 -05:00
|
|
|
it("preserves existing result fields when merging stop metadata", () => {
|
|
|
|
|
const result = mergeHeartbeatRunStopMetadata(
|
|
|
|
|
{ summary: "done" },
|
|
|
|
|
buildHeartbeatRunStopMetadata({
|
|
|
|
|
adapterType: "openclaw_gateway",
|
|
|
|
|
adapterConfig: {},
|
|
|
|
|
outcome: "succeeded",
|
|
|
|
|
}),
|
|
|
|
|
);
|
|
|
|
|
|
|
|
|
|
expect(result).toMatchObject({
|
|
|
|
|
summary: "done",
|
|
|
|
|
stopReason: "completed",
|
|
|
|
|
effectiveTimeoutSec: 120,
|
|
|
|
|
timeoutConfigured: true,
|
|
|
|
|
timeoutSource: "default",
|
|
|
|
|
timeoutFired: false,
|
|
|
|
|
});
|
|
|
|
|
});
|
|
|
|
|
});
|