[codex] Harden heartbeat scheduling and runtime controls (#4223)

## Thinking Path

> - Paperclip orchestrates AI agents through issue checkout, heartbeat
runs, routines, and auditable control-plane state
> - The runtime path has to recover from lost local processes, transient
adapter failures, blocked dependencies, and routine coalescing without
stranding work
> - The existing branch carried several reliability fixes across
heartbeat scheduling, issue runtime controls, routine dispatch, and
operator-facing run state
> - These changes belong together because they share backend contracts,
migrations, and runtime status semantics
> - This pull request groups the control-plane/runtime slice so it can
merge independently from board UI polish and adapter sandbox work
> - The benefit is safer heartbeat recovery, clearer runtime controls,
and more predictable recurring execution behavior

## What Changed

- Adds bounded heartbeat retry scheduling, scheduled retry state, and
Codex transient failure recovery handling.
- Tightens heartbeat process recovery, blocker wake behavior, issue
comment wake handling, routine dispatch coalescing, and
activity/dashboard bounds.
- Adds runtime-control MCP tools and Paperclip skill docs for issue
workspace runtime management.
- Adds migrations `0061_lively_thor_girl.sql` and
`0062_routine_run_dispatch_fingerprint.sql`.
- Surfaces retry state in run ledger/agent UI and keeps related shared
types synchronized.

## Verification

- `pnpm exec vitest run
server/src/__tests__/heartbeat-retry-scheduling.test.ts
server/src/__tests__/heartbeat-process-recovery.test.ts
server/src/__tests__/routines-service.test.ts`
- `pnpm exec vitest run src/tools.test.ts` from `packages/mcp-server`

## Risks

- Medium risk: this touches heartbeat recovery and routine dispatch,
which are central execution paths.
- Migration order matters if split branches land out of order: merge
this PR before branches that assume the new runtime/routine fields.
- Runtime retry behavior should be watched in CI and in local operator
smoke tests because it changes how transient failures are resumed.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- OpenAI Codex, GPT-5-based coding agent runtime, shell/git tool use
enabled. Exact hosted model build and context window are not exposed in
this Paperclip heartbeat environment.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] If this change affects the UI, I have included before/after
screenshots
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] I will address all Greptile and reviewer comments before
requesting merge

packages/mcp-server/README.md

@@ -47,6 +47,8 @@ Read tools:
 - `paperclipListDocumentRevisions`
 - `paperclipListProjects`
 - `paperclipGetProject`
+- `paperclipGetIssueWorkspaceRuntime`
+- `paperclipWaitForIssueWorkspaceService`
 - `paperclipListGoals`
 - `paperclipGetGoal`
 - `paperclipListApprovals`
@@ -63,6 +65,7 @@ Write tools:
 - `paperclipAddComment`
 - `paperclipUpsertIssueDocument`
 - `paperclipRestoreIssueDocumentRevision`
+- `paperclipControlIssueWorkspaceServices`
 - `paperclipCreateApproval`
 - `paperclipLinkIssueApproval`
 - `paperclipUnlinkIssueApproval`

packages/mcp-server/src/tools.test.ts

@@ -107,6 +107,81 @@ describe("paperclip MCP tools", () => {
     });
   });
 
+  it("controls issue workspace services through the current execution workspace", async () => {
+    const fetchMock = vi.fn()
+      .mockResolvedValueOnce(mockJsonResponse({
+        currentExecutionWorkspace: {
+          id: "44444444-4444-4444-8444-444444444444",
+          runtimeServices: [],
+        },
+      }))
+      .mockResolvedValueOnce(mockJsonResponse({
+        operation: { id: "operation-1" },
+        workspace: {
+          id: "44444444-4444-4444-8444-444444444444",
+          runtimeServices: [
+            {
+              id: "55555555-5555-4555-8555-555555555555",
+              serviceName: "web",
+              status: "running",
+              url: "http://127.0.0.1:5173",
+            },
+          ],
+        },
+      }));
+    vi.stubGlobal("fetch", fetchMock);
+
+    const tool = getTool("paperclipControlIssueWorkspaceServices");
+    await tool.execute({
+      issueId: "PAP-1135",
+      action: "restart",
+      workspaceCommandId: "web",
+    });
+
+    expect(fetchMock).toHaveBeenCalledTimes(2);
+    const [lookupUrl, lookupInit] = fetchMock.mock.calls[0] as [string, RequestInit];
+    expect(String(lookupUrl)).toBe("http://localhost:3100/api/issues/PAP-1135/heartbeat-context");
+    expect(lookupInit.method).toBe("GET");
+
+    const [controlUrl, controlInit] = fetchMock.mock.calls[1] as [string, RequestInit];
+    expect(String(controlUrl)).toBe(
+      "http://localhost:3100/api/execution-workspaces/44444444-4444-4444-8444-444444444444/runtime-services/restart",
+    );
+    expect(controlInit.method).toBe("POST");
+    expect(JSON.parse(String(controlInit.body))).toEqual({
+      workspaceCommandId: "web",
+    });
+  });
+
+  it("waits for an issue workspace runtime service URL", async () => {
+    const fetchMock = vi.fn()
+      .mockResolvedValueOnce(mockJsonResponse({
+        currentExecutionWorkspace: {
+          id: "44444444-4444-4444-8444-444444444444",
+          runtimeServices: [
+            {
+              id: "55555555-5555-4555-8555-555555555555",
+              serviceName: "web",
+              status: "running",
+              healthStatus: "healthy",
+              url: "http://127.0.0.1:5173",
+            },
+          ],
+        },
+      }));
+    vi.stubGlobal("fetch", fetchMock);
+
+    const tool = getTool("paperclipWaitForIssueWorkspaceService");
+    const response = await tool.execute({
+      issueId: "PAP-1135",
+      serviceName: "web",
+      timeoutSeconds: 1,
+    });
+
+    expect(fetchMock).toHaveBeenCalledTimes(1);
+    expect(response.content[0]?.text).toContain("http://127.0.0.1:5173");
+  });
+
   it("creates approvals with the expected company-scoped payload", async () => {
     const fetchMock = vi.fn().mockResolvedValue(
       mockJsonResponse({ id: "approval-1" }),

packages/mcp-server/src/tools.ts

@@ -124,6 +124,66 @@ const apiRequestSchema = z.object({
   jsonBody: z.string().optional(),
 });
 
+const workspaceRuntimeControlTargetSchema = z.object({
+  workspaceCommandId: z.string().min(1).optional().nullable(),
+  runtimeServiceId: z.string().uuid().optional().nullable(),
+  serviceIndex: z.number().int().nonnegative().optional().nullable(),
+});
+
+const issueWorkspaceRuntimeControlSchema = z.object({
+  issueId: issueIdSchema,
+  action: z.enum(["start", "stop", "restart"]),
+}).merge(workspaceRuntimeControlTargetSchema);
+
+const waitForIssueWorkspaceServiceSchema = z.object({
+  issueId: issueIdSchema,
+  runtimeServiceId: z.string().uuid().optional().nullable(),
+  serviceName: z.string().min(1).optional().nullable(),
+  timeoutSeconds: z.number().int().positive().max(300).optional(),
+});
+
+function sleep(ms: number) {
+  return new Promise((resolve) => setTimeout(resolve, ms));
+}
+
+function readCurrentExecutionWorkspace(context: unknown): Record<string, unknown> | null {
+  if (!context || typeof context !== "object") return null;
+  const workspace = (context as { currentExecutionWorkspace?: unknown }).currentExecutionWorkspace;
+  return workspace && typeof workspace === "object" ? workspace as Record<string, unknown> : null;
+}
+
+function readWorkspaceRuntimeServices(workspace: Record<string, unknown> | null): Array<Record<string, unknown>> {
+  const raw = workspace?.runtimeServices;
+  return Array.isArray(raw)
+    ? raw.filter((entry): entry is Record<string, unknown> => Boolean(entry) && typeof entry === "object")
+    : [];
+}
+
+function selectRuntimeService(
+  services: Array<Record<string, unknown>>,
+  input: { runtimeServiceId?: string | null; serviceName?: string | null },
+) {
+  if (input.runtimeServiceId) {
+    return services.find((service) => service.id === input.runtimeServiceId) ?? null;
+  }
+  if (input.serviceName) {
+    return services.find((service) => service.serviceName === input.serviceName) ?? null;
+  }
+  return services.find((service) => service.status === "running" || service.status === "starting")
+    ?? services[0]
+    ?? null;
+}
+
+async function getIssueWorkspaceRuntime(client: PaperclipApiClient, issueId: string) {
+  const context = await client.requestJson("GET", `/issues/${encodeURIComponent(issueId)}/heartbeat-context`);
+  const workspace = readCurrentExecutionWorkspace(context);
+  return {
+    context,
+    workspace,
+    runtimeServices: readWorkspaceRuntimeServices(workspace),
+  };
+}
+
 export function createToolDefinitions(client: PaperclipApiClient): ToolDefinition[] {
   return [
     makeTool(
@@ -247,6 +307,55 @@ export function createToolDefinitions(client: PaperclipApiClient): ToolDefinitio
         return client.requestJson("GET", `/projects/${encodeURIComponent(projectId)}${qs}`);
       },
     ),
+    makeTool(
+      "paperclipGetIssueWorkspaceRuntime",
+      "Get the current execution workspace and runtime services for an issue, including service URLs",
+      z.object({ issueId: issueIdSchema }),
+      async ({ issueId }) => getIssueWorkspaceRuntime(client, issueId),
+    ),
+    makeTool(
+      "paperclipControlIssueWorkspaceServices",
+      "Start, stop, or restart the current issue execution workspace runtime services",
+      issueWorkspaceRuntimeControlSchema,
+      async ({ issueId, action, ...target }) => {
+        const runtime = await getIssueWorkspaceRuntime(client, issueId);
+        const workspaceId = typeof runtime.workspace?.id === "string" ? runtime.workspace.id : null;
+        if (!workspaceId) {
+          throw new Error("Issue has no current execution workspace");
+        }
+        return client.requestJson(
+          "POST",
+          `/execution-workspaces/${encodeURIComponent(workspaceId)}/runtime-services/${action}`,
+          { body: target },
+        );
+      },
+    ),
+    makeTool(
+      "paperclipWaitForIssueWorkspaceService",
+      "Wait until an issue execution workspace runtime service is running and has a URL when one is exposed",
+      waitForIssueWorkspaceServiceSchema,
+      async ({ issueId, runtimeServiceId, serviceName, timeoutSeconds }) => {
+        const deadline = Date.now() + (timeoutSeconds ?? 60) * 1000;
+        let latest: Awaited<ReturnType<typeof getIssueWorkspaceRuntime>> | null = null;
+        while (Date.now() <= deadline) {
+          latest = await getIssueWorkspaceRuntime(client, issueId);
+          const service = selectRuntimeService(latest.runtimeServices, { runtimeServiceId, serviceName });
+          if (service?.status === "running" && service.healthStatus !== "unhealthy") {
+            return {
+              workspace: latest.workspace,
+              service,
+            };
+          }
+          await sleep(1000);
+        }
+
+        return {
+          timedOut: true,
+          latestWorkspace: latest?.workspace ?? null,
+          latestRuntimeServices: latest?.runtimeServices ?? [],
+        };
+      },
+    ),
     makeTool(
       "paperclipListGoals",
       "List goals in a company",