mirror of
https://github.com/alkimake/paperclip.git
synced 2026-06-17 19:20:39 +09:00
[codex] Harden recovery issue handling (#4600)
## Thinking Path > - Paperclip orchestrates AI agents for zero-human companies > - The control plane must recover stranded agent work without creating new operational loops > - Stranded recovery issues can themselves fail, and exposing raw retry errors in comments can leak sensitive adapter details > - New local companies also should not force a hire-approval gate unless operators enable that policy > - This pull request hardens recovery issue handling, redacts retry failure details in issue copy, preserves `maxConcurrentRuns: 1`, and flips new-hire approval to an opt-in default > - The benefit is safer automatic recovery and smoother default company setup without hidden migration conflicts ## What Changed - Added migration `0071_default_hire_approval_off` and updated company schema/import/export/docs so hire approvals default off and serialize only when enabled. - Added migration `0072_large_sandman` with a partial unique index preventing duplicate active stranded recovery issues for the same source issue. - Blocked failed `stranded_issue_recovery` issues in place instead of creating nested recovery issues. - Redacted latest retry failure details from recovery issue comments while still linking reviewers to run evidence. - Allowed `maxConcurrentRuns: 1` to be honored by heartbeat concurrency normalization. - Added focused regression coverage for recovery recursion, redaction, migration ordering, and concurrency behavior. ## Verification - `pnpm --filter @paperclipai/db run check:migrations` - `pnpm exec vitest run --project @paperclipai/server server/src/__tests__/recovery-classifiers.test.ts` - `pnpm exec vitest run --project @paperclipai/server server/src/__tests__/company-portability.test.ts --pool=forks --poolOptions.forks.isolate=true` - `pnpm exec vitest run --project @paperclipai/server server/src/__tests__/agent-permissions-routes.test.ts --pool=forks --poolOptions.forks.isolate=true` - `pnpm --filter @paperclipai/server typecheck` - `pnpm exec vitest run --project @paperclipai/server server/src/__tests__/heartbeat-process-recovery.test.ts --pool=forks --poolOptions.forks.isolate=true` exits 0, but this host skipped the embedded Postgres tests with the existing init guard. - `pnpm exec vitest run --project @paperclipai/server server/src/__tests__/heartbeat-dependency-scheduling.test.ts --pool=forks --poolOptions.forks.isolate=true` exits 0, but this host skipped the embedded Postgres tests with the existing init guard. ## Risks - Migration risk is low but this PR intentionally owns both new migrations to avoid separate PR migration-journal conflicts. - Recovery comments now require operators to inspect linked run evidence for details instead of reading raw errors inline. - The hire approval default changes behavior for newly created/imported companies only; existing persisted company settings are not changed except by the SQL default for future rows. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI Codex, GPT-5 coding agent, tool-enabled terminal/GitHub workflow, reasoning mode active. Context window not exposed in this environment. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] If this change affects the UI, I have included before/after screenshots - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
parent
6ccf80bcf2
commit
7a9b3a6037
18 changed files with 16535 additions and 65 deletions
|
|
@ -96,7 +96,16 @@ describeEmbeddedPostgres("heartbeat dependency-aware queued run selection", () =
|
|||
}, 20_000);
|
||||
|
||||
afterEach(async () => {
|
||||
vi.clearAllMocks();
|
||||
mockAdapterExecute.mockReset();
|
||||
mockAdapterExecute.mockImplementation(async () => ({
|
||||
exitCode: 0,
|
||||
signal: null,
|
||||
timedOut: false,
|
||||
errorMessage: null,
|
||||
summary: "Dependency-aware heartbeat test run.",
|
||||
provider: "test",
|
||||
model: "test-model",
|
||||
}));
|
||||
runningProcesses.clear();
|
||||
let idlePolls = 0;
|
||||
for (let attempt = 0; attempt < 100; attempt += 1) {
|
||||
|
|
@ -347,6 +356,126 @@ describeEmbeddedPostgres("heartbeat dependency-aware queued run selection", () =
|
|||
expect(blockedWakeRequestCount).toBeGreaterThanOrEqual(2);
|
||||
});
|
||||
|
||||
it("honors maxConcurrentRuns 1 by leaving a second assignment wake queued", async () => {
|
||||
const companyId = randomUUID();
|
||||
const agentId = randomUUID();
|
||||
const firstIssueId = randomUUID();
|
||||
const secondIssueId = randomUUID();
|
||||
let finishFirstRun!: () => void;
|
||||
const firstRunFinished = new Promise<void>((resolve) => {
|
||||
finishFirstRun = resolve;
|
||||
});
|
||||
|
||||
mockAdapterExecute.mockImplementationOnce(async () => {
|
||||
await firstRunFinished;
|
||||
return {
|
||||
exitCode: 0,
|
||||
signal: null,
|
||||
timedOut: false,
|
||||
errorMessage: null,
|
||||
summary: "First assignment run completed.",
|
||||
provider: "test",
|
||||
model: "test-model",
|
||||
};
|
||||
});
|
||||
|
||||
await db.insert(companies).values({
|
||||
id: companyId,
|
||||
name: "Paperclip",
|
||||
issuePrefix: `T${companyId.replace(/-/g, "").slice(0, 6).toUpperCase()}`,
|
||||
requireBoardApprovalForNewAgents: false,
|
||||
});
|
||||
await db.insert(agents).values({
|
||||
id: agentId,
|
||||
companyId,
|
||||
name: "CodexCoder",
|
||||
role: "engineer",
|
||||
status: "active",
|
||||
adapterType: "codex_local",
|
||||
adapterConfig: {},
|
||||
runtimeConfig: {
|
||||
heartbeat: {
|
||||
wakeOnDemand: true,
|
||||
maxConcurrentRuns: 1,
|
||||
},
|
||||
},
|
||||
permissions: {},
|
||||
});
|
||||
await db.insert(issues).values([
|
||||
{
|
||||
id: firstIssueId,
|
||||
companyId,
|
||||
title: "First assignment",
|
||||
status: "todo",
|
||||
priority: "high",
|
||||
assigneeAgentId: agentId,
|
||||
},
|
||||
{
|
||||
id: secondIssueId,
|
||||
companyId,
|
||||
title: "Second assignment",
|
||||
status: "todo",
|
||||
priority: "high",
|
||||
assigneeAgentId: agentId,
|
||||
},
|
||||
]);
|
||||
|
||||
try {
|
||||
const firstWake = await heartbeat.wakeup(agentId, {
|
||||
source: "assignment",
|
||||
triggerDetail: "system",
|
||||
reason: "issue_assigned",
|
||||
payload: { issueId: firstIssueId },
|
||||
contextSnapshot: { issueId: firstIssueId, wakeReason: "issue_assigned" },
|
||||
});
|
||||
expect(firstWake).not.toBeNull();
|
||||
|
||||
const firstRunStarted = await waitForCondition(async () => {
|
||||
const run = await db
|
||||
.select({ status: heartbeatRuns.status })
|
||||
.from(heartbeatRuns)
|
||||
.where(eq(heartbeatRuns.id, firstWake!.id))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
return run?.status === "running";
|
||||
});
|
||||
expect(firstRunStarted).toBe(true);
|
||||
const firstAdapterStarted = await waitForCondition(async () => mockAdapterExecute.mock.calls.length === 1);
|
||||
expect(firstAdapterStarted).toBe(true);
|
||||
|
||||
const secondWake = await heartbeat.wakeup(agentId, {
|
||||
source: "assignment",
|
||||
triggerDetail: "system",
|
||||
reason: "issue_assigned",
|
||||
payload: { issueId: secondIssueId },
|
||||
contextSnapshot: { issueId: secondIssueId, wakeReason: "issue_assigned" },
|
||||
});
|
||||
expect(secondWake).not.toBeNull();
|
||||
|
||||
const secondRunWhileFirstRunning = await db
|
||||
.select({ status: heartbeatRuns.status })
|
||||
.from(heartbeatRuns)
|
||||
.where(eq(heartbeatRuns.id, secondWake!.id))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
expect(secondRunWhileFirstRunning?.status).toBe("queued");
|
||||
expect(mockAdapterExecute).toHaveBeenCalledTimes(1);
|
||||
|
||||
finishFirstRun();
|
||||
|
||||
const secondRunSucceeded = await waitForCondition(async () => {
|
||||
const run = await db
|
||||
.select({ status: heartbeatRuns.status })
|
||||
.from(heartbeatRuns)
|
||||
.where(eq(heartbeatRuns.id, secondWake!.id))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
return run?.status === "succeeded";
|
||||
});
|
||||
expect(secondRunSucceeded).toBe(true);
|
||||
expect(mockAdapterExecute).toHaveBeenCalledTimes(2);
|
||||
} finally {
|
||||
finishFirstRun();
|
||||
}
|
||||
});
|
||||
|
||||
it("cancels stale queued runs when issue blockers are still unresolved", async () => {
|
||||
const companyId = randomUUID();
|
||||
const agentId = randomUUID();
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue