[codex] Harden execution reliability and heartbeat tooling (#3679)

## Thinking Path

> - Paperclip orchestrates AI agents for zero-human companies
> - Reliable execution depends on heartbeat routing, issue lifecycle
semantics, telemetry, and a fast enough local verification loop to keep
regressions visible
> - The remaining commits on this branch were mostly server/runtime
correctness fixes plus test and documentation follow-ups in that area
> - Those changes are logically separate from the UI-focused
issue-detail and workspace/navigation branches even when they touch
overlapping issue APIs
> - This pull request groups the execution reliability, heartbeat,
telemetry, and tooling changes into one standalone branch
> - The benefit is a focused review of the control-plane correctness
work, including the follow-up fix that restored the implicit
comment-reopen helpers after branch splitting

## What Changed

- Hardened issue/heartbeat execution behavior, including self-review
stage skipping, deferred mention wakes during active execution, stranded
execution recovery, active-run scoping, assignee resolution, and
blocked-to-todo wake resumption
- Reduced noisy polling/logging overhead by trimming issue run payloads,
compacting persisted run logs, silencing high-volume request logs, and
capping heartbeat-run queries in dashboard/inbox surfaces
- Expanded telemetry and status semantics with adapter/model fields on
task completion plus clearer status guidance in docs/onboarding material
- Updated test infrastructure and verification defaults with faster
route-test module isolation, cheaper default `pnpm test`, e2e isolation
from local state, and repo verification follow-ups
- Included docs/release housekeeping from the branch and added a small
follow-up commit restoring the implicit comment-reopen helpers that were
dropped during branch reconstruction

## Verification

- `pnpm vitest run
server/src/__tests__/issue-comment-reopen-routes.test.ts
server/src/__tests__/issue-telemetry-routes.test.ts`
- `pnpm vitest run server/src/__tests__/http-log-policy.test.ts
server/src/__tests__/heartbeat-run-log.test.ts
server/src/__tests__/health.test.ts`
- `server/src/__tests__/activity-service.test.ts`,
`server/src/__tests__/heartbeat-comment-wake-batching.test.ts`, and
`server/src/__tests__/heartbeat-process-recovery.test.ts` were attempted
on this host but the embedded Postgres harness reported
init-script/data-dir problems and skipped or failed to start, so they
are noted as environment-limited

## Risks

- Medium: this branch changes core issue/heartbeat routing and
reopen/wakeup behavior, so regressions would affect agent execution flow
rather than isolated UI polish
- Because it also updates verification infrastructure, reviewers should
pay attention to whether the new tests are asserting the right failure
modes and not just reshaping harness behavior

## Model Used

- OpenAI Codex coding agent (GPT-5-class runtime in Codex CLI; exact
deployed model ID is not exposed in this environment), reasoning
enabled, tool use and local code execution enabled

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] If this change affects the UI, I have included before/after
screenshots
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

releases/v2026.413.0.md

@@ -0,0 +1,98 @@
+# v2026.413.0
+
+> Released: 2026-04-13
+
+## Highlights
+
+- **Issue chat thread** — Replaced the classic comment timeline with a full chat-style thread powered by assistant-ui. Agent run transcripts, chain-of-thought, and user messages now render inline as a continuous conversation with polished avatars, action bars, and relative timestamps. ([#3079](https://github.com/paperclipai/paperclip/pull/3079))
+- **External adapter plugin system** — Third-party adapters can now be installed as npm packages or loaded from local directories. Plugins declare a config schema and an optional UI transcript parser; built-in adapters can be overridden by external ones. Includes Hermes local session management and provider/model display in run details. ([#2649](https://github.com/paperclipai/paperclip/pull/2649), [#2650](https://github.com/paperclipai/paperclip/pull/2650), [#2651](https://github.com/paperclipai/paperclip/pull/2651), [#2654](https://github.com/paperclipai/paperclip/pull/2654), [#2655](https://github.com/paperclipai/paperclip/pull/2655), [#2659](https://github.com/paperclipai/paperclip/pull/2659), @plind-dm)
+- **Execution policies** — Issues can now carry a review/approval execution policy with multi-stage signoff workflows. Reviewers and approvers are selected per-stage, and Paperclip routes the issue through each stage automatically. ([#3222](https://github.com/paperclipai/paperclip/pull/3222))
+- **Blocker dependencies** — First-class issue blocker relations with automatic wake-on-dependency-resolved. Set `blockedByIssueIds` on any issue and Paperclip wakes the assignee when all blockers reach `done`. ([#2797](https://github.com/paperclipai/paperclip/pull/2797))
+- **Standalone MCP server** — New `@paperclipai/mcp-server` package exposing the Paperclip API as an MCP tool server, including approval creation. ([#2435](https://github.com/paperclipai/paperclip/pull/2435))
+
+## Improvements
+
+- **Board approvals** — Generic issue-linked board approvals with card styling and visibility improvements in the issue detail sidebar. ([#3220](https://github.com/paperclipai/paperclip/pull/3220))
+- **Inbox parent-child nesting** — Parent issues group their children in the inbox Mine view with a toggle button, j/k keyboard traversal across nested items, and collapsible groups. ([#2218](https://github.com/paperclipai/paperclip/pull/2218), @HenkDz)
+- **Inbox workspace grouping** — Issues can now be grouped by workspace in the inbox with collapsible mobile groups and shared column controls across inbox and issues lists. ([#3356](https://github.com/paperclipai/paperclip/pull/3356))
+- **Issue search** — Trigram-indexed full-text search across titles, identifiers, descriptions, and comments with debounced input. Comment matches now surface in search results. ([#2999](https://github.com/paperclipai/paperclip/pull/2999))
+- **Sub-issues inline** — Sub-issues moved from a separate tab to inline display on the issue detail, with parent-inherited workspace defaults and assignee propagation. ([#3355](https://github.com/paperclipai/paperclip/pull/3355))
+- **Document revision diff viewer** — Side-by-side diff viewer for issue document revisions with improved modal layout. ([#2792](https://github.com/paperclipai/paperclip/pull/2792))
+- **Keyboard shortcuts cheatsheet** — Press `?` to open a keyboard shortcut reference dialog; new `g i` (go to inbox), `g c` (comment composer), and inbox archive undo shortcuts. ([#2772](https://github.com/paperclipai/paperclip/pull/2772))
+- **Bedrock model selection** — Claude local adapter now supports AWS Bedrock authentication and model selection. ([#3033](https://github.com/paperclipai/paperclip/pull/3033), @kimnamu)
+- **Codex fast mode** — Added fast mode support for the Codex local adapter. ([#3383](https://github.com/paperclipai/paperclip/pull/3383))
+- **Backup improvements** — Gzip-compressed backups with tiered daily/weekly/monthly retention and UI controls in Instance Settings. ([#3015](https://github.com/paperclipai/paperclip/pull/3015), @aronprins)
+- **GitHub webhook signing modes** — Added `github_hmac` and `none` webhook signing modes with timing-safe HMAC comparison. ([#1961](https://github.com/paperclipai/paperclip/pull/1961), @antonio-mello-ai)
+- **Project environment variables** — Projects can now define environment variables that are inherited by workspace runs.
+- **Routine improvements** — Draft routine defaults, run-time overrides, routine title variables, and relaxed project/agent requirements for routines. ([#3220](https://github.com/paperclipai/paperclip/pull/3220))
+- **Workspace runtime controls** — Start/stop controls, runtime state reconciliation, runtime service JSON textarea improvements, and workspace branch/folder display in the issue properties sidebar. ([#3354](https://github.com/paperclipai/paperclip/pull/3354))
+- **Attachment improvements** — Arbitrary file attachments (not just images), drag-and-drop non-image files onto markdown editor, and square-cropped image gallery grid. ([#2749](https://github.com/paperclipai/paperclip/pull/2749))
+- **Image gallery in chat** — Clicking images in chat messages now opens a full gallery viewer.
+- **Mobile UX** — Gmail-inspired mobile top bar for inbox issue views, responsive execution workspace pages, mobile mention menu placement, and mobile comment copy button feedback.
+- **Sidebar order persistence** — Sidebar project and company ordering preferences now persist per-user.
+- **Skill auto-enable** — Mentioned skills are automatically enabled for heartbeat runs.
+- **Comment wake batching** — Multiple comment wakes are batched into a single inline payload for more efficient agent heartbeats.
+- **Server-side adapter pause/resume** — Builtin adapter types can now be paused/resumed from the server with `overridePaused`. ([#2542](https://github.com/paperclipai/paperclip/pull/2542), @plind-dm)
+- **Skill slash-command autocomplete** — Skill names now autocomplete in the editor.
+- **Worktree reseed command** — New CLI command to reseed worktrees from latest repo state. ([#3353](https://github.com/paperclipai/paperclip/pull/3353))
+
+## Fixes
+
+- **Issue detail stability** — Fixed visible refreshes during agent updates, comment post resets, ref update loops, split regressions, and main-pane focus on navigation. ([#3355](https://github.com/paperclipai/paperclip/pull/3355))
+- **Inbox badge count** — Badge now correctly counts only unread Mine issues. ([#2512](https://github.com/paperclipai/paperclip/pull/2512), @AllenHyang)
+- **Inbox keyboard navigation** — Fixed j/k traversal across groups and nesting column alignment. ([#2218](https://github.com/paperclipai/paperclip/pull/2218), @HenkDz)
+- **Vite HTML transforms** — Fixed repeated vite HTML transforms in dev mode.
+- **Auth session lookup** — Skipped unnecessary auth session lookups on non-API requests.
+- **Stale execution locks** — Fixed stale execution lock lifecycle with proper `executionAgentNameKey` clearing. ([#2643](https://github.com/paperclipai/paperclip/pull/2643), @chrisschwer)
+- **Agent env bindings** — Fixed cleared agent env bindings not persisting on save. ([#3232](https://github.com/paperclipai/paperclip/pull/3232), @officialasishkumar)
+- **Capabilities field** — Fixed blank screen when clearing the Capabilities field. ([#2442](https://github.com/paperclipai/paperclip/pull/2442), @sparkeros)
+- **Skill deletion** — Company skills can now be deleted with an agent usage check. ([#2441](https://github.com/paperclipai/paperclip/pull/2441), @DanielSousa)
+- **Claude session resume** — Fixed `--append-system-prompt-file` being sent on resumed Claude sessions and preserved instructions on resume fallback. ([#2949](https://github.com/paperclipai/paperclip/pull/2949), [#2936](https://github.com/paperclipai/paperclip/pull/2936), [#2937](https://github.com/paperclipai/paperclip/pull/2937), @Lempkey)
+- **JWT secret fallback** — Removed hardcoded JWT secret fallback; auth now properly falls back to `BETTER_AUTH_SECRET`. ([#3124](https://github.com/paperclipai/paperclip/pull/3124), @cleanunicorn)
+- **Agent auth JWT** — Fixed agent auth to fall back to `BETTER_AUTH_SECRET` when `PAPERCLIP_AGENT_JWT_SECRET` is absent. ([#2866](https://github.com/paperclipai/paperclip/pull/2866), @ergonaworks)
+- **Typing lag** — Fixed typing lag in long comment threads. ([#3163](https://github.com/paperclipai/paperclip/pull/3163))
+- **Infinite render loop** — Fixed infinite render loop in inbox mobile toolbar.
+- **Shimmer animation** — Fixed shimmer text using invalid `hsl()` wrapper on `oklch` colors, loop jitter, and added pause between repeats.
+- **Mention selection** — Restored touch mention selection and fixed spaced mention queries.
+- **Inbox archive** — Fixed archive flashing back after fade-out.
+- **Goal description** — Made goal description area scrollable in create dialog. ([#2148](https://github.com/paperclipai/paperclip/pull/2148), @shoaib050326)
+- **Worktree provisioning** — Fixed symlink relinking, fallback seeding, dependency hydration, and validated linked worktrees before reuse. ([#3354](https://github.com/paperclipai/paperclip/pull/3354))
+- **Node keepAliveTimeout** — Increased timeout behind reverse proxies to prevent 502 errors.
+- **Noisy request logging** — Reduced noisy server request logging.
+- **Codex tool-use transcripts** — Fixed Codex tool-use transcript completion parsing.
+- **Codex resume error** — Recognize missing-rollout Codex resume error as stale session.
+- **Pi quota exhaustion** — Treat Pi quota exhaustion as a failed run. ([#2305](https://github.com/paperclipai/paperclip/pull/2305))
+- **Security** — Bumped rollup to 4.59.0 (path-traversal CVE), multer to 2.1.1 (HIGH CVEs), and redacted Bearer tokens from server log output. ([#2909](https://github.com/paperclipai/paperclip/pull/2909), @marysomething99-prog)
+- **Issue identifier collisions** — Prevented identifier collisions during concurrent issue creation.
+- **OpenClaw CEO paths** — Fixed `$AGENT_HOME` references in CEO onboarding instructions to use relative paths. ([#3299](https://github.com/paperclipai/paperclip/pull/3299), @aronprins)
+- **Route authorization** — Scoped import, approvals, activity, and heartbeat routes properly. ([#3009](https://github.com/paperclipai/paperclip/pull/3009), @KhairulA)
+- **Windows adapter** — Uses `cmd.exe` for `.cmd`/`.bat` wrappers on Windows. ([#2662](https://github.com/paperclipai/paperclip/pull/2662), @wbelt)
+- **Markdown autoformat** — Fixed autoformat of pasted markdown in inline editor. ([#2733](https://github.com/paperclipai/paperclip/pull/2733), @davison)
+- **Paused agent dimming** — Correctly dim paused agents in list and org chart views; skip dimming on Paused filter tab. ([#2397](https://github.com/paperclipai/paperclip/pull/2397), @HearthCore)
+- **Import role fallback** — Import now reads agent role from frontmatter before defaulting to "agent". ([#2594](https://github.com/paperclipai/paperclip/pull/2594), @plind-dm)
+- **Backup cleanup** — Clean up orphaned `.sql` files on compression failure and fix stale startup log.
+
+## Upgrade Guide
+
+Eight new database migrations (`0049`–`0056`) will run automatically on startup. These add:
+
+- Issue blocker relations table (`0049`)
+- Project environment variables (`0050`)
+- Trigram search indexes on issues and comments (`0051` — requires `pg_trgm` extension)
+- Execution policy decision tracking (`0052`)
+- Non-issue inbox dismissals (`0053`)
+- Relaxed routine constraints (`0054`)
+- Heartbeat run process group tracking (`0055`)
+- User sidebar preferences (`0056`)
+
+All migrations are additive — no existing data is modified or removed.
+
+**`pg_trgm` extension**: Migration `0051` creates the `pg_trgm` PostgreSQL extension for full-text search. If your database user does not have `CREATE EXTENSION` privileges, ask your DBA to run `CREATE EXTENSION IF NOT EXISTS pg_trgm;` before upgrading.
+
+If you use external adapter plugins, note that built-in adapters can now be overridden by external ones. The `overriddenBuiltin` flag in the adapter API indicates when this is happening.
+
+## Contributors
+
+Thank you to everyone who contributed to this release!
+
+@AllenHyang, @antonio-mello-ai, @aronprins, @chrisschwer, @cleanunicorn, @cryppadotta, @DanielSousa, @davison, @ergonaworks, @HearthCore, @HenkDz, @KhairulA, @kimnamu, @Lempkey, @marysomething99-prog, @mvanhorn, @officialasishkumar, @plind-dm, @shoaib050326, @sparkeros, @wbelt