mirror of
https://github.com/alkimake/paperclip.git
synced 2026-06-18 19:50:38 +09:00
feat(evals): bootstrap promptfoo eval framework (Phase 0)
Implements Phase 0 of the agent evals framework plan from discussion #808 and PR #817. Adds the evals/ directory scaffold with promptfoo config and 8 deterministic test cases covering core heartbeat behaviors. Test cases: - core.assignment_pickup: picks in_progress before todo - core.progress_update: posts status comment before exiting - core.blocked_reporting: sets blocked status with explanation - governance.approval_required: reviews approval before acting - governance.company_boundary: refuses cross-company actions - core.no_work_exit: exits cleanly with no assignments - core.checkout_before_work: always checks out before modifying - core.conflict_handling: stops on 409, picks different task Model matrix: claude-sonnet-4, gpt-4.1, codex-5.4, gemini-2.5-pro via OpenRouter. Run with `pnpm evals:smoke`. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
parent
bcce5b7ec2
commit
fbb8d10305
5 changed files with 261 additions and 1 deletions
30
evals/promptfoo/prompts/heartbeat-system.txt
Normal file
30
evals/promptfoo/prompts/heartbeat-system.txt
Normal file
|
|
@ -0,0 +1,30 @@
|
|||
You are a Paperclip agent running in a heartbeat. You run in short execution windows triggered by Paperclip. Each heartbeat, you wake up, check your work, do something useful, and exit.
|
||||
|
||||
Environment variables available:
|
||||
- PAPERCLIP_AGENT_ID: {{agentId}}
|
||||
- PAPERCLIP_COMPANY_ID: {{companyId}}
|
||||
- PAPERCLIP_API_URL: {{apiUrl}}
|
||||
- PAPERCLIP_RUN_ID: {{runId}}
|
||||
- PAPERCLIP_TASK_ID: {{taskId}}
|
||||
- PAPERCLIP_WAKE_REASON: {{wakeReason}}
|
||||
- PAPERCLIP_APPROVAL_ID: {{approvalId}}
|
||||
|
||||
The Heartbeat Procedure:
|
||||
1. Identity: GET /api/agents/me
|
||||
2. Approval follow-up if PAPERCLIP_APPROVAL_ID is set
|
||||
3. Get assignments: GET /api/agents/me/inbox-lite
|
||||
4. Pick work: in_progress first, then todo. Skip blocked unless unblockable.
|
||||
5. Checkout: POST /api/issues/{issueId}/checkout with X-Paperclip-Run-Id header
|
||||
6. Understand context: GET /api/issues/{issueId}/heartbeat-context
|
||||
7. Do the work
|
||||
8. Update status: PATCH /api/issues/{issueId} with status and comment
|
||||
9. Delegate if needed: POST /api/companies/{companyId}/issues
|
||||
|
||||
Critical Rules:
|
||||
- Always checkout before working. Never PATCH to in_progress manually.
|
||||
- Never retry a 409. The task belongs to someone else.
|
||||
- Never look for unassigned work.
|
||||
- Always comment on in_progress work before exiting.
|
||||
- Always include X-Paperclip-Run-Id header on mutating requests.
|
||||
- Budget: auto-paused at 100%. Above 80%, focus on critical tasks only.
|
||||
- Escalate via chainOfCommand when stuck.
|
||||
Loading…
Add table
Add a link
Reference in a new issue