mirror of
https://github.com/alkimake/paperclip.git
synced 2026-06-14 01:50:39 +09:00
Fixes #6470 ## Thinking Path > - Paperclip is an open-source AI agent platform receiving a high volume of community PRs — currently 2,398 open > - The contributor experience is broken: PRs sit for months with no feedback, contributors don't know why they're stuck, and maintainers spend review time on PRs that are missing basics > - Common problems: no linked issue, no test coverage, incomplete PR template, manually-edited lockfile — all catchable before human review > - At the same time, accepting untrusted PRs from unknown contributors is a real attack surface: malicious packages, secret injection, tampering with CI scripts, and code touching the sensitive paths from the April security advisories > - This PR adds automated gates that run on every PR: quality failures get a clear comment telling contributors exactly what to fix, security concerns are silently flagged as draft advisories and block merge via a pending check run > - The benefit is a dramatically faster feedback loop for good-faith contributors and a meaningful security layer for the maintainers reviewing them ## What Changed - **`.github/workflows/commitperclip-review.yml`** — new workflow using `pull_request_target` (runs in base branch context, has secrets, never executes PR code). Runs quality gates + security gates on every PR open/update. - **`.github/dependabot.yml`** — weekly automated dependency vulnerability PRs for npm and GitHub Actions. - **`.github/scripts/get-bot-token.mjs`** — generates a short-lived commitperclip installation token from `COMMITPERCLIP_KEY` secret. - **`.github/scripts/run-quality-gates.mjs`** — orchestrates 5 quality gates, posts/updates a single consolidated comment on the PR. - **`.github/scripts/check-pr-template.mjs`** — validates all 5 required template sections, Thinking Path depth (≥3 sentences), Model Used not placeholder. - **`.github/scripts/check-pr-linked-issue.mjs`** — requires `Fixes #NNN` or issue URL in PR body. - **`.github/scripts/check-pr-test-coverage.mjs`** — requires at least one test file in the diff. - **`.github/scripts/check-pr-lockfile.mjs`** — blocks manual `pnpm-lock.yaml` edits (only the refresh bot may change it). - **`.github/scripts/check-pr-dependencies.mjs`** — informational comment when new npm packages are added. - **`.github/scripts/check-pr-security.mjs`** — 6 silent security checks: secret patterns, CI workflow tampering, build script changes, supply chain (new packages in lockfile), suspicious test patterns (outbound network/shell exec/env var reads), and changes to the 9 sensitive path prefixes from the April advisories. When any fire: creates a draft security advisory + sets `security-review` check to `in_progress` (blocks merge). When clean: sets `security-review` to `success`. - **`actions/dependency-review-action@v4`** — per-PR dependency vulnerability check (fails if new dep has known CVE). - **44 unit tests** across all gate modules (`node:test`, no external deps). ## Verification Run all unit tests locally: ```bash node --test .github/scripts/tests/*.test.mjs ``` Expected: 44 pass, 0 fail. End-to-end: open a PR missing the template, linked issue, and test files → commitperclip posts a consolidated comment listing all failures. Open a PR with all gates satisfied → `✅ All checks passing` comment posted, all check runs green. ## Risks **`pull_request_target` security model:** This workflow runs in base branch context and has access to secrets. It explicitly checks out `ref: master` (never PR code) and reads the PR diff via GitHub API only — no PR code is ever executed. This is the correct pattern for running secret-bearing checks on fork PRs; deviating from it (e.g. checking out the PR branch) would be a security vulnerability. **False positives on security gates:** The sensitive-path gate flags any PR touching the 9 path prefixes from the April advisories. Legitimate fixes to those paths will trigger draft advisories. This is intentional — those paths warrant a human look regardless. The `security-review` check can be manually resolved by a maintainer once reviewed. **commitperclip not yet installed:** Until the app is installed on this repo and the `COMMITPERCLIP_KEY` secret is added, the workflow will fail on the token generation step. The quality gate comment won't post, but Dependency Review will still run independently. ## Model Used Claude Sonnet 4.5, 200k context window, extended thinking enabled, tool use: read/edit files, bash execution, GitHub API calls ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have run tests locally and they pass (44/44) - [x] I have added or updated tests where applicable (44 unit tests across all gate modules) - [ ] If this change affects the UI, I have included before/after screenshots (N/A — CI only) - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] I will address all Greptile and reviewer comments before requesting merge --- ## One-time setup needed from you, Dotta 1. **Install commitperclip app** on this repo: https://github.com/apps/commitperclip/installations/new 2. **Add `COMMITPERCLIP_KEY`** as a repository secret (Actions → Secrets) — ask @brandonburr for the key 3. **Add `security_advisories: write` and `checks: write`** to the commitperclip app permissions (commit-capital org → Settings → Apps → commitperclip → Permissions) 4. **Install Socket.dev** from GitHub Marketplace for supply chain scanning 5. **Branch protection** (optional but recommended): require `commitperclip-review` and `security-review` checks to pass before merge ## Dashboard integration note The `commitperclip-review` check run result maps cleanly to your PR triage dashboard. A single filter on your Worker: ```javascript const gatesCheck = checkRuns.find(r => r.name === 'commitperclip-review'); if (gatesCheck?.conclusion === 'failure') return null; // filter from queue ``` For security flags: `GET /repos/paperclipai/paperclip/security-advisories?state=draft` — advisory titles include `PR #NNN` for cross-referencing. PRs with a matching draft advisory have `security-review` in `in_progress` state (grey spinner, can't merge via branch protection). --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Devin Foley <devin@devinfoley.com> Co-authored-by: Paperclip <noreply@paperclip.ing>
111 lines
3.7 KiB
JavaScript
111 lines
3.7 KiB
JavaScript
import { test } from 'node:test';
|
|
import assert from 'node:assert/strict';
|
|
import { checkLinkedIssue } from '../check-pr-linked-issue.mjs';
|
|
|
|
// Existing tests with title parameter added (defaults to no prefix, so still required)
|
|
|
|
test('passes with bare #NNN reference', () => {
|
|
assert.equal(checkLinkedIssue('This fixes the bug in #123', 'fix: something').passed, true);
|
|
});
|
|
|
|
test('passes with "Fixes #NNN"', () => {
|
|
assert.equal(checkLinkedIssue('Fixes #456\n\nSome description', 'fix: something').passed, true);
|
|
});
|
|
|
|
test('passes with "Closes #NNN" (case-insensitive)', () => {
|
|
assert.equal(checkLinkedIssue('closes #789', 'fix: something').passed, true);
|
|
});
|
|
|
|
test('passes with "Resolves #NNN"', () => {
|
|
assert.equal(checkLinkedIssue('Resolves #101', 'fix: something').passed, true);
|
|
});
|
|
|
|
test('passes with full github.com URL', () => {
|
|
assert.equal(
|
|
checkLinkedIssue('See https://github.com/paperclipai/paperclip/issues/202', 'fix: bug').passed,
|
|
true
|
|
);
|
|
});
|
|
|
|
test('passes with a full github.com URL followed by punctuation', () => {
|
|
assert.equal(
|
|
checkLinkedIssue('See (https://github.com/paperclipai/paperclip/issues/202).', 'fix: bug').passed,
|
|
true
|
|
);
|
|
});
|
|
|
|
test('fails with empty body when no skip prefix', () => {
|
|
const result = checkLinkedIssue('', 'fix: bug');
|
|
assert.equal(result.passed, false);
|
|
assert.ok(result.failures.length > 0);
|
|
});
|
|
|
|
test('fails with no issue reference when no skip prefix', () => {
|
|
const result = checkLinkedIssue('Added a cool feature, no issue linked', 'feat: something');
|
|
assert.equal(result.passed, false);
|
|
assert.ok(result.failures[0].includes('Fixes #NNN'));
|
|
});
|
|
|
|
test('fails with cross-repo issue reference', () => {
|
|
const result = checkLinkedIssue('See https://github.com/other/repo/issues/123', 'fix: bug');
|
|
assert.equal(result.passed, false);
|
|
});
|
|
|
|
test('fails when the Paperclip issue URL is embedded inside another host', () => {
|
|
const result = checkLinkedIssue(
|
|
'See https://evil.example/https://github.com/paperclipai/paperclip/issues/123',
|
|
'fix: bug'
|
|
);
|
|
assert.equal(result.passed, false);
|
|
});
|
|
|
|
test('fails when the Paperclip issue URL continues into another host', () => {
|
|
const result = checkLinkedIssue(
|
|
'See https://github.com/paperclipai/paperclip/issues/123.evil.example',
|
|
'fix: bug'
|
|
);
|
|
assert.equal(result.passed, false);
|
|
});
|
|
|
|
test('fails when #NNN is part of a word (no space before)', () => {
|
|
const result = checkLinkedIssue('This is version#123 not an issue link', 'fix: bug');
|
|
assert.equal(result.passed, false);
|
|
});
|
|
|
|
// New tests for prefix-aware skip behavior
|
|
|
|
test('skips check for docs: prefix', () => {
|
|
assert.equal(checkLinkedIssue('', 'docs: update README').passed, true);
|
|
});
|
|
|
|
test('skips check for chore: prefix', () => {
|
|
assert.equal(checkLinkedIssue('', 'chore: bump deps').passed, true);
|
|
});
|
|
|
|
test('skips check for build: prefix', () => {
|
|
assert.equal(checkLinkedIssue('', 'build: update Dockerfile').passed, true);
|
|
});
|
|
|
|
test('skips check for ci: prefix', () => {
|
|
assert.equal(checkLinkedIssue('', 'ci: add workflow').passed, true);
|
|
});
|
|
|
|
test('skips check for test: prefix', () => {
|
|
assert.equal(checkLinkedIssue('', 'test: add coverage').passed, true);
|
|
});
|
|
|
|
test('skips check with scoped prefix like docs(api):', () => {
|
|
assert.equal(checkLinkedIssue('', 'docs(api): document endpoint').passed, true);
|
|
});
|
|
|
|
test('requires issue for feat: prefix', () => {
|
|
assert.equal(checkLinkedIssue('Some description without issue', 'feat: new thing').passed, false);
|
|
});
|
|
|
|
test('requires issue for refactor: prefix', () => {
|
|
assert.equal(checkLinkedIssue('Some refactor', 'refactor: rewrite thing').passed, false);
|
|
});
|
|
|
|
test('requires issue when no prefix (encourages prefix usage)', () => {
|
|
assert.equal(checkLinkedIssue('No prefix here', 'Add some feature').passed, false);
|
|
});
|