ake/paperclip
1
0
Fork 0
mirror of https://github.com/alkimake/paperclip.git synced 2026-06-14 01:50:39 +09:00
Code Issues Projects Releases Packages Wiki Activity
paperclip/evals/promptfoo/promptfooconfig.yaml

37 lines
1,019 B
YAML
Raw Normal View History

feat(evals): bootstrap promptfoo eval framework (Phase 0) Implements Phase 0 of the agent evals framework plan from discussion #808 and PR #817. Adds the evals/ directory scaffold with promptfoo config and 8 deterministic test cases covering core heartbeat behaviors. Test cases: - core.assignment_pickup: picks in_progress before todo - core.progress_update: posts status comment before exiting - core.blocked_reporting: sets blocked status with explanation - governance.approval_required: reviews approval before acting - governance.company_boundary: refuses cross-company actions - core.no_work_exit: exits cleanly with no assignments - core.checkout_before_work: always checks out before modifying - core.conflict_handling: stops on 409, picks different task Model matrix: claude-sonnet-4, gpt-4.1, codex-5.4, gemini-2.5-pro via OpenRouter. Run with `pnpm evals:smoke`. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-03-13 17:09:51 -07:00
# Paperclip Agent Evals - Phase 0: Promptfoo Bootstrap
#
# Tests narrow heartbeat behaviors across models with deterministic assertions.
refactor(evals): split test cases into tests/*.yaml files Move inline test cases from promptfooconfig.yaml into separate files organized by category (core.yaml, governance.yaml). Main config now uses file://tests/*.yaml glob pattern per promptfoo best practices. This makes it easier to add new test categories without bloating the main config, and lets contributors add cases by dropping new YAML files into tests/.
2026-03-15 12:15:51 -07:00
# Test cases are organized by category in tests/*.yaml files.
feat(evals): bootstrap promptfoo eval framework (Phase 0) Implements Phase 0 of the agent evals framework plan from discussion #808 and PR #817. Adds the evals/ directory scaffold with promptfoo config and 8 deterministic test cases covering core heartbeat behaviors. Test cases: - core.assignment_pickup: picks in_progress before todo - core.progress_update: posts status comment before exiting - core.blocked_reporting: sets blocked status with explanation - governance.approval_required: reviews approval before acting - governance.company_boundary: refuses cross-company actions - core.no_work_exit: exits cleanly with no assignments - core.checkout_before_work: always checks out before modifying - core.conflict_handling: stops on 409, picks different task Model matrix: claude-sonnet-4, gpt-4.1, codex-5.4, gemini-2.5-pro via OpenRouter. Run with `pnpm evals:smoke`. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-03-13 17:09:51 -07:00
# See doc/plans/2026-03-13-agent-evals-framework.md for the full framework plan.
#
# Usage:
# cd evals/promptfoo && promptfoo eval
# promptfoo view # open results in browser
#
refactor(evals): split test cases into tests/*.yaml files Move inline test cases from promptfooconfig.yaml into separate files organized by category (core.yaml, governance.yaml). Main config now uses file://tests/*.yaml glob pattern per promptfoo best practices. This makes it easier to add new test categories without bloating the main config, and lets contributors add cases by dropping new YAML files into tests/.
2026-03-15 12:15:51 -07:00
# Validate config before committing:
# promptfoo validate
#
feat(evals): bootstrap promptfoo eval framework (Phase 0) Implements Phase 0 of the agent evals framework plan from discussion #808 and PR #817. Adds the evals/ directory scaffold with promptfoo config and 8 deterministic test cases covering core heartbeat behaviors. Test cases: - core.assignment_pickup: picks in_progress before todo - core.progress_update: posts status comment before exiting - core.blocked_reporting: sets blocked status with explanation - governance.approval_required: reviews approval before acting - governance.company_boundary: refuses cross-company actions - core.no_work_exit: exits cleanly with no assignments - core.checkout_before_work: always checks out before modifying - core.conflict_handling: stops on 409, picks different task Model matrix: claude-sonnet-4, gpt-4.1, codex-5.4, gemini-2.5-pro via OpenRouter. Run with `pnpm evals:smoke`. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-03-13 17:09:51 -07:00
# Requires OPENROUTER_API_KEY or individual provider keys.
description: "Paperclip heartbeat behavior evals"
prompts:
- file://prompts/heartbeat-system.txt
providers:
- id: openrouter:anthropic/claude-sonnet-4-20250514
label: claude-sonnet-4
- id: openrouter:openai/gpt-4.1
label: gpt-4.1
- id: openrouter:openai/codex-5.4
label: codex-5.4
- id: openrouter:google/gemini-2.5-pro
label: gemini-2.5-pro
defaultTest:
options:
transformVars: "{ ...vars, apiUrl: 'http://localhost:18080', runId: 'run-eval-001' }"
tests:
refactor(evals): split test cases into tests/*.yaml files Move inline test cases from promptfooconfig.yaml into separate files organized by category (core.yaml, governance.yaml). Main config now uses file://tests/*.yaml glob pattern per promptfoo best practices. This makes it easier to add new test categories without bloating the main config, and lets contributors add cases by dropping new YAML files into tests/.
2026-03-15 12:15:51 -07:00
- file://tests/*.yaml
Reference in a new issue Copy permalink