[codex] Harden heartbeat scheduling and runtime controls (#4223)

## Thinking Path

> - Paperclip orchestrates AI agents through issue checkout, heartbeat
runs, routines, and auditable control-plane state
> - The runtime path has to recover from lost local processes, transient
adapter failures, blocked dependencies, and routine coalescing without
stranding work
> - The existing branch carried several reliability fixes across
heartbeat scheduling, issue runtime controls, routine dispatch, and
operator-facing run state
> - These changes belong together because they share backend contracts,
migrations, and runtime status semantics
> - This pull request groups the control-plane/runtime slice so it can
merge independently from board UI polish and adapter sandbox work
> - The benefit is safer heartbeat recovery, clearer runtime controls,
and more predictable recurring execution behavior

## What Changed

- Adds bounded heartbeat retry scheduling, scheduled retry state, and
Codex transient failure recovery handling.
- Tightens heartbeat process recovery, blocker wake behavior, issue
comment wake handling, routine dispatch coalescing, and
activity/dashboard bounds.
- Adds runtime-control MCP tools and Paperclip skill docs for issue
workspace runtime management.
- Adds migrations `0061_lively_thor_girl.sql` and
`0062_routine_run_dispatch_fingerprint.sql`.
- Surfaces retry state in run ledger/agent UI and keeps related shared
types synchronized.

## Verification

- `pnpm exec vitest run
server/src/__tests__/heartbeat-retry-scheduling.test.ts
server/src/__tests__/heartbeat-process-recovery.test.ts
server/src/__tests__/routines-service.test.ts`
- `pnpm exec vitest run src/tools.test.ts` from `packages/mcp-server`

## Risks

- Medium risk: this touches heartbeat recovery and routine dispatch,
which are central execution paths.
- Migration order matters if split branches land out of order: merge
this PR before branches that assume the new runtime/routine fields.
- Runtime retry behavior should be watched in CI and in local operator
smoke tests because it changes how transient failures are resumed.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- OpenAI Codex, GPT-5-based coding agent runtime, shell/git tool use
enabled. Exact hosted model build and context window are not exposed in
this Paperclip heartbeat environment.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] If this change affects the UI, I have included before/after
screenshots
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] I will address all Greptile and reviewer comments before
requesting merge
This commit is contained in:
Dotta 2026-04-21 12:24:11 -05:00 committed by GitHub
parent ab9051b595
commit 09d0678840
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
61 changed files with 17622 additions and 456 deletions

View file

@ -43,6 +43,7 @@ import { PackageFileTree, buildFileTree } from "../components/PackageFileTree";
import { ScrollToBottom } from "../components/ScrollToBottom";
import { formatCents, formatDate, relativeTime, formatTokens, visibleRunCostUsd } from "../lib/utils";
import { cn } from "../lib/utils";
import { describeRunRetryState } from "../lib/runRetryState";
import { Button } from "@/components/ui/button";
import { Skeleton } from "@/components/ui/skeleton";
import { Tabs } from "@/components/ui/tabs";
@ -104,6 +105,7 @@ const runStatusIcons: Record<string, { icon: typeof CheckCircle2; color: string
failed: { icon: XCircle, color: "text-red-600 dark:text-red-400" },
running: { icon: Loader2, color: "text-cyan-600 dark:text-cyan-400" },
queued: { icon: Clock, color: "text-yellow-600 dark:text-yellow-400" },
scheduled_retry: { icon: Clock, color: "text-sky-600 dark:text-sky-400" },
timed_out: { icon: Timer, color: "text-orange-600 dark:text-orange-400" },
cancelled: { icon: Slash, color: "text-neutral-500 dark:text-neutral-400" },
};
@ -2342,26 +2344,39 @@ function PromptsTab({
</p>
</div>
</div>
{selectedFileExists && !selectedFileSummary?.deprecated && selectedOrEntryFile !== currentEntryFile && (
<Button
type="button"
size="sm"
variant="outline"
onClick={() => {
if (confirm(`Delete ${selectedOrEntryFile}?`)) {
deleteFile.mutate(selectedOrEntryFile, {
onSuccess: () => {
setSelectedFile(currentEntryFile);
setDraft(null);
},
});
}
}}
disabled={deleteFile.isPending}
>
Delete
</Button>
)}
<div className="flex items-center gap-2">
{!fileLoading && (
<CopyText
text={displayValue}
ariaLabel="Copy instructions file as markdown"
title="Copy as markdown"
copiedLabel="Copied"
className="inline-flex h-8 w-8 items-center justify-center rounded-md border border-border text-muted-foreground hover:bg-accent hover:text-foreground"
>
<Copy className="h-3.5 w-3.5" />
</CopyText>
)}
{selectedFileExists && !selectedFileSummary?.deprecated && selectedOrEntryFile !== currentEntryFile && (
<Button
type="button"
size="sm"
variant="outline"
onClick={() => {
if (confirm(`Delete ${selectedOrEntryFile}?`)) {
deleteFile.mutate(selectedOrEntryFile, {
onSuccess: () => {
setSelectedFile(currentEntryFile);
setDraft(null);
},
});
}
}}
disabled={deleteFile.isPending}
>
Delete
</Button>
)}
</div>
</div>
{selectedFileExists && fileLoading && !selectedFileDetail ? (
@ -3141,6 +3156,7 @@ function RunDetail({ run: initialRun, agentRouteId, adapterType, adapterConfig }
const sessionChanged = run.sessionIdBefore && run.sessionIdAfter && run.sessionIdBefore !== run.sessionIdAfter;
const sessionId = run.sessionIdAfter || run.sessionIdBefore;
const hasNonZeroExit = run.exitCode !== null && run.exitCode !== 0;
const retryState = describeRunRetryState(run);
return (
<div className="space-y-4 min-w-0">
@ -3295,6 +3311,30 @@ function RunDetail({ run: initialRun, agentRouteId, adapterType, adapterConfig }
{run.signal && <span className="text-muted-foreground ml-1">(signal: {run.signal})</span>}
</div>
)}
{retryState && (
<div className="rounded-md border border-border/70 bg-accent/20 px-3 py-2 text-xs leading-5">
<div className="flex flex-wrap items-center gap-2">
<span
className={cn(
"rounded-md border px-1.5 py-0.5 text-[11px] font-medium",
retryState.tone,
)}
>
{retryState.badgeLabel}
</span>
{retryState.retryOfRunId ? (
<Link
to={`/agents/${agentRouteId}/runs/${retryState.retryOfRunId}`}
className="font-mono text-foreground hover:underline"
>
{retryState.retryOfRunId.slice(0, 8)}
</Link>
) : null}
</div>
{retryState.detail ? <p className="mt-2 text-muted-foreground">{retryState.detail}</p> : null}
{retryState.secondary ? <p className="text-muted-foreground">{retryState.secondary}</p> : null}
</div>
)}
</div>
{/* Right column: metrics */}