From 96feaa331ab8d398cf420873d6085b69b69f6b85 Mon Sep 17 00:00:00 2001 From: brandonburr Date: Mon, 1 Jun 2026 10:52:53 -0600 Subject: [PATCH] feat(commitperclip): add automated PR quality and security gates (#6469) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fixes #6470 ## Thinking Path > - Paperclip is an open-source AI agent platform receiving a high volume of community PRs — currently 2,398 open > - The contributor experience is broken: PRs sit for months with no feedback, contributors don't know why they're stuck, and maintainers spend review time on PRs that are missing basics > - Common problems: no linked issue, no test coverage, incomplete PR template, manually-edited lockfile — all catchable before human review > - At the same time, accepting untrusted PRs from unknown contributors is a real attack surface: malicious packages, secret injection, tampering with CI scripts, and code touching the sensitive paths from the April security advisories > - This PR adds automated gates that run on every PR: quality failures get a clear comment telling contributors exactly what to fix, security concerns are silently flagged as draft advisories and block merge via a pending check run > - The benefit is a dramatically faster feedback loop for good-faith contributors and a meaningful security layer for the maintainers reviewing them ## What Changed - **`.github/workflows/commitperclip-review.yml`** — new workflow using `pull_request_target` (runs in base branch context, has secrets, never executes PR code). Runs quality gates + security gates on every PR open/update. - **`.github/dependabot.yml`** — weekly automated dependency vulnerability PRs for npm and GitHub Actions. - **`.github/scripts/get-bot-token.mjs`** — generates a short-lived commitperclip installation token from `COMMITPERCLIP_KEY` secret. - **`.github/scripts/run-quality-gates.mjs`** — orchestrates 5 quality gates, posts/updates a single consolidated comment on the PR. - **`.github/scripts/check-pr-template.mjs`** — validates all 5 required template sections, Thinking Path depth (≥3 sentences), Model Used not placeholder. - **`.github/scripts/check-pr-linked-issue.mjs`** — requires `Fixes #NNN` or issue URL in PR body. - **`.github/scripts/check-pr-test-coverage.mjs`** — requires at least one test file in the diff. - **`.github/scripts/check-pr-lockfile.mjs`** — blocks manual `pnpm-lock.yaml` edits (only the refresh bot may change it). - **`.github/scripts/check-pr-dependencies.mjs`** — informational comment when new npm packages are added. - **`.github/scripts/check-pr-security.mjs`** — 6 silent security checks: secret patterns, CI workflow tampering, build script changes, supply chain (new packages in lockfile), suspicious test patterns (outbound network/shell exec/env var reads), and changes to the 9 sensitive path prefixes from the April advisories. When any fire: creates a draft security advisory + sets `security-review` check to `in_progress` (blocks merge). When clean: sets `security-review` to `success`. - **`actions/dependency-review-action@v4`** — per-PR dependency vulnerability check (fails if new dep has known CVE). - **44 unit tests** across all gate modules (`node:test`, no external deps). ## Verification Run all unit tests locally: ```bash node --test .github/scripts/tests/*.test.mjs ``` Expected: 44 pass, 0 fail. End-to-end: open a PR missing the template, linked issue, and test files → commitperclip posts a consolidated comment listing all failures. Open a PR with all gates satisfied → `✅ All checks passing` comment posted, all check runs green. ## Risks **`pull_request_target` security model:** This workflow runs in base branch context and has access to secrets. It explicitly checks out `ref: master` (never PR code) and reads the PR diff via GitHub API only — no PR code is ever executed. This is the correct pattern for running secret-bearing checks on fork PRs; deviating from it (e.g. checking out the PR branch) would be a security vulnerability. **False positives on security gates:** The sensitive-path gate flags any PR touching the 9 path prefixes from the April advisories. Legitimate fixes to those paths will trigger draft advisories. This is intentional — those paths warrant a human look regardless. The `security-review` check can be manually resolved by a maintainer once reviewed. **commitperclip not yet installed:** Until the app is installed on this repo and the `COMMITPERCLIP_KEY` secret is added, the workflow will fail on the token generation step. The quality gate comment won't post, but Dependency Review will still run independently. ## Model Used Claude Sonnet 4.5, 200k context window, extended thinking enabled, tool use: read/edit files, bash execution, GitHub API calls ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have run tests locally and they pass (44/44) - [x] I have added or updated tests where applicable (44 unit tests across all gate modules) - [ ] If this change affects the UI, I have included before/after screenshots (N/A — CI only) - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] I will address all Greptile and reviewer comments before requesting merge --- ## One-time setup needed from you, Dotta 1. **Install commitperclip app** on this repo: https://github.com/apps/commitperclip/installations/new 2. **Add `COMMITPERCLIP_KEY`** as a repository secret (Actions → Secrets) — ask @brandonburr for the key 3. **Add `security_advisories: write` and `checks: write`** to the commitperclip app permissions (commit-capital org → Settings → Apps → commitperclip → Permissions) 4. **Install Socket.dev** from GitHub Marketplace for supply chain scanning 5. **Branch protection** (optional but recommended): require `commitperclip-review` and `security-review` checks to pass before merge ## Dashboard integration note The `commitperclip-review` check run result maps cleanly to your PR triage dashboard. A single filter on your Worker: ```javascript const gatesCheck = checkRuns.find(r => r.name === 'commitperclip-review'); if (gatesCheck?.conclusion === 'failure') return null; // filter from queue ``` For security flags: `GET /repos/paperclipai/paperclip/security-advisories?state=draft` — advisory titles include `PR #NNN` for cross-referencing. PRs with a matching draft advisory have `security-review` in `in_progress` state (grey spinner, can't merge via branch protection). --------- Co-authored-by: Claude Sonnet 4.6 Co-authored-by: Devin Foley Co-authored-by: Paperclip --- .github/dependabot.yml | 23 ++ .github/scripts/check-pr-dependencies.mjs | 89 +++++ .github/scripts/check-pr-linked-issue.mjs | 53 +++ .github/scripts/check-pr-lockfile.mjs | 31 ++ .github/scripts/check-pr-security.mjs | 350 ++++++++++++++++++ .github/scripts/check-pr-template.mjs | 88 +++++ .github/scripts/check-pr-test-coverage.mjs | 83 +++++ .github/scripts/fetch-pr-files.mjs | 21 ++ .github/scripts/get-bot-token.mjs | 113 ++++++ .github/scripts/run-quality-gates.mjs | 145 ++++++++ .../tests/check-pr-dependencies.test.mjs | 65 ++++ .../tests/check-pr-linked-issue.test.mjs | 111 ++++++ .../scripts/tests/check-pr-lockfile.test.mjs | 33 ++ .../scripts/tests/check-pr-security.test.mjs | 316 ++++++++++++++++ .../scripts/tests/check-pr-template.test.mjs | 123 ++++++ .../tests/check-pr-test-coverage.test.mjs | 101 +++++ .github/scripts/tests/fetch-pr-files.test.mjs | 37 ++ .github/scripts/tests/get-bot-token.test.mjs | 33 ++ .../scripts/tests/run-quality-gates.test.mjs | 43 +++ .github/workflows/commitperclip-review.yml | 69 ++++ .gitignore | 2 + 21 files changed, 1929 insertions(+) create mode 100644 .github/dependabot.yml create mode 100644 .github/scripts/check-pr-dependencies.mjs create mode 100644 .github/scripts/check-pr-linked-issue.mjs create mode 100644 .github/scripts/check-pr-lockfile.mjs create mode 100644 .github/scripts/check-pr-security.mjs create mode 100644 .github/scripts/check-pr-template.mjs create mode 100644 .github/scripts/check-pr-test-coverage.mjs create mode 100644 .github/scripts/fetch-pr-files.mjs create mode 100644 .github/scripts/get-bot-token.mjs create mode 100644 .github/scripts/run-quality-gates.mjs create mode 100644 .github/scripts/tests/check-pr-dependencies.test.mjs create mode 100644 .github/scripts/tests/check-pr-linked-issue.test.mjs create mode 100644 .github/scripts/tests/check-pr-lockfile.test.mjs create mode 100644 .github/scripts/tests/check-pr-security.test.mjs create mode 100644 .github/scripts/tests/check-pr-template.test.mjs create mode 100644 .github/scripts/tests/check-pr-test-coverage.test.mjs create mode 100644 .github/scripts/tests/fetch-pr-files.test.mjs create mode 100644 .github/scripts/tests/get-bot-token.test.mjs create mode 100644 .github/scripts/tests/run-quality-gates.test.mjs create mode 100644 .github/workflows/commitperclip-review.yml diff --git a/.github/dependabot.yml b/.github/dependabot.yml new file mode 100644 index 00000000..ff0b3514 --- /dev/null +++ b/.github/dependabot.yml @@ -0,0 +1,23 @@ +version: 2 +updates: + - package-ecosystem: npm + directory: "/" + schedule: + interval: weekly + day: monday + time: "06:00" + open-pull-requests-limit: 10 + labels: + - "dependencies" + ignore: + # Ignore major version bumps — review those manually + - dependency-name: "*" + update-types: ["version-update:semver-major"] + + - package-ecosystem: github-actions + directory: "/" + schedule: + interval: weekly + day: monday + time: "06:00" + open-pull-requests-limit: 5 diff --git a/.github/scripts/check-pr-dependencies.mjs b/.github/scripts/check-pr-dependencies.mjs new file mode 100644 index 00000000..c31ed3e4 --- /dev/null +++ b/.github/scripts/check-pr-dependencies.mjs @@ -0,0 +1,89 @@ +#!/usr/bin/env node +/** + * check-pr-dependencies.mjs + * Detects new npm packages added in this PR vs. the base branch. + * Never fails (informational only) — outputs { passed: true, informational: string[] } + */ +import { fileURLToPath } from 'node:url'; +import { ghFetch } from './get-bot-token.mjs'; + +function buildContentsPath(repo, filename, ref) { + return `/repos/${repo}/contents/${filename}?${new URLSearchParams({ ref }).toString()}`; +} + +export async function resolveBaseRef(fetchFromGitHub, token, repo, prNumber, baseRef) { + if (baseRef) { + return baseRef; + } + + const pr = await fetchFromGitHub(`/repos/${repo}/pulls/${prNumber}`, token); + if (pr.base?.ref) { + return pr.base.ref; + } + + const repository = await fetchFromGitHub(`/repos/${repo}`, token); + if (repository.default_branch) { + return repository.default_branch; + } + + throw new Error(`Unable to resolve a base branch for ${repo}#${prNumber}.`); +} + +export async function checkDependencies(files, token, repo, prNumber, baseRef, fetchFromGitHub = ghFetch) { + const pkgFiles = files.filter( + f => f.filename.endsWith('package.json') && + !f.filename.includes('node_modules') && + f.status !== 'removed' + ); + + if (pkgFiles.length === 0) return { passed: true, informational: [] }; + + const resolvedBaseRef = await resolveBaseRef(fetchFromGitHub, token, repo, prNumber, baseRef); + const newPackages = new Set(); + + for (const file of pkgFiles) { + try { + const [baseRes, prRes] = await Promise.all([ + fetchFromGitHub(buildContentsPath(repo, file.filename, resolvedBaseRef), token), + fetchFromGitHub(buildContentsPath(repo, file.filename, `refs/pull/${prNumber}/head`), token), + ]); + + const basePkg = JSON.parse(Buffer.from(baseRes.content, 'base64').toString()); + const prPkg = JSON.parse(Buffer.from(prRes.content, 'base64').toString()); + + const baseDeps = new Set([ + ...Object.keys(basePkg.dependencies ?? {}), + ...Object.keys(basePkg.devDependencies ?? {}), + ...Object.keys(basePkg.peerDependencies ?? {}), + ]); + + for (const dep of [ + ...Object.keys(prPkg.dependencies ?? {}), + ...Object.keys(prPkg.devDependencies ?? {}), + ...Object.keys(prPkg.peerDependencies ?? {}), + ]) { + if (!baseDeps.has(dep)) newPackages.add(dep); + } + } catch { + // File may not exist on base — skip + } + } + + if (newPackages.size === 0) return { passed: true, informational: [] }; + + const pkgList = [...newPackages].map(p => `\`${p}\``).join(', '); + return { + passed: true, + informational: [ + `📦 New dependencies added: ${pkgList}. Review may take longer and new dependencies ` + + `are less likely to be accepted — please check if existing deps cover this need.`, + ], + }; +} + +if (process.argv[1] === fileURLToPath(import.meta.url)) { + const { GH_TOKEN, GH_REPO, PR_NUMBER, PR_FILES, PR_BASE_REF } = process.env; + const files = JSON.parse(PR_FILES ?? '[]'); + const result = await checkDependencies(files, GH_TOKEN, GH_REPO, PR_NUMBER, PR_BASE_REF); + console.log(JSON.stringify(result)); +} diff --git a/.github/scripts/check-pr-linked-issue.mjs b/.github/scripts/check-pr-linked-issue.mjs new file mode 100644 index 00000000..865b8865 --- /dev/null +++ b/.github/scripts/check-pr-linked-issue.mjs @@ -0,0 +1,53 @@ +#!/usr/bin/env node +/** + * check-pr-linked-issue.mjs + * Checks that a PR body references a GitHub issue. Respects conventional commit + * prefixes — skips check for docs/chore/build/ci/style/test prefixed PRs. + * Export: checkLinkedIssue(prBody: string, prTitle: string) → { passed, failures } + */ +import { fileURLToPath } from 'node:url'; + +const ISSUE_PATTERNS = [ + /(?:fixes|closes|resolves)\s+#\d+/i, + /(?:^|[\s(])https:\/\/github\.com\/paperclipai\/paperclip\/issues\/\d+(?=$|[\s),:;!?]|[.](?![\w-]))/i, + /(? p.test(body)); + return { + passed: found, + failures: found ? [] : [ + 'No linked issue found — please add `Fixes #NNN` to your PR description. ' + + 'If no issue exists yet, please file one first: ' + + 'https://github.com/paperclipai/paperclip/issues/new', + ], + }; +} + +if (process.argv[1] === fileURLToPath(import.meta.url)) { + const body = process.env.PR_BODY ?? ''; + const title = process.env.PR_TITLE ?? ''; + const result = checkLinkedIssue(body, title); + console.log(JSON.stringify(result)); + process.exit(result.passed ? 0 : 1); +} diff --git a/.github/scripts/check-pr-lockfile.mjs b/.github/scripts/check-pr-lockfile.mjs new file mode 100644 index 00000000..e8352006 --- /dev/null +++ b/.github/scripts/check-pr-lockfile.mjs @@ -0,0 +1,31 @@ +#!/usr/bin/env node +/** + * check-pr-lockfile.mjs + * Checks that pnpm-lock.yaml was not manually edited. + * Export: checkLockfile(files, prAuthor, prBranch) → { passed, failures } + */ +import { fileURLToPath } from 'node:url'; + +export function checkLockfile(files, prAuthor, prBranch) { + const lockfileChanged = files.some(f => f.filename === 'pnpm-lock.yaml'); + if (!lockfileChanged) return { passed: true, failures: [] }; + + const isRefreshBot = + prAuthor === 'github-actions[bot]' && prBranch === 'chore/refresh-lockfile'; + + return { + passed: isRefreshBot, + failures: isRefreshBot ? [] : [ + 'You have changes to `pnpm-lock.yaml` — `pr.yml` will hard-fail this PR with a confusing message about lockfile edits. ' + + 'To fix: run `pnpm install` locally, exclude the lockfile from your commit, push again. ' + + 'The lockfile is regenerated automatically by the refresh bot on a schedule.', + ], + }; +} + +if (process.argv[1] === fileURLToPath(import.meta.url)) { + const files = JSON.parse(process.env.PR_FILES ?? '[]'); + const result = checkLockfile(files, process.env.PR_AUTHOR ?? '', process.env.PR_BRANCH ?? ''); + console.log(JSON.stringify(result)); + process.exit(result.passed ? 0 : 1); +} diff --git a/.github/scripts/check-pr-security.mjs b/.github/scripts/check-pr-security.mjs new file mode 100644 index 00000000..8d3ac9ab --- /dev/null +++ b/.github/scripts/check-pr-security.mjs @@ -0,0 +1,350 @@ +#!/usr/bin/env node +/** + * check-pr-security.mjs + * Runs 6 security checks against a PR diff. Never posts public comments. + * Creates a draft security advisory in the repo if any check fires. + * + * Env: GH_TOKEN, GH_REPO, PR_NUMBER, PR_AUTHOR + * Exit: always 0 — security flags are silent, never block the PR visibly. + */ +import { fileURLToPath } from 'node:url'; +import { ghFetch } from './get-bot-token.mjs'; +import { fetchAllPullRequestFiles } from './fetch-pr-files.mjs'; +import { resolveBaseRef } from './check-pr-dependencies.mjs'; + +// ── Pure check functions (exported for testing) ─────────────────────────────── + +const SECRET_PATTERNS = [ + { name: 'OpenAI API key', re: /sk-[a-zA-Z0-9]{32,}/ }, + { name: 'Google API key', re: /AIza[0-9A-Za-z\-_]{35}/ }, + { name: 'AWS access key', re: /AKIA[0-9A-Z]{16}/ }, + { name: 'Private key', re: /-----BEGIN (RSA|EC|OPENSSH) PRIVATE KEY-----/ }, + { name: 'High-entropy secret', re: /[a-zA-Z_]*(key|token|secret|password|credential)[a-zA-Z_]*\s*[=:]\s*["'][^"']{20,}["']/i }, +]; + +export function scanSecrets(files) { + const flags = []; + for (const file of files) { + if (!file.patch) continue; + const added = file.patch.split('\n').filter(l => l.startsWith('+') && !l.startsWith('+++')); + for (const line of added) { + for (const { name, re } of SECRET_PATTERNS) { + if (re.test(line)) { + flags.push({ check: 'secret-scan', file: file.filename, pattern: name, line: line.slice(0, 120) }); + } + } + } + } + return flags; +} + +const CI_BUILD_SCRIPTS = [ + 'scripts/release.sh', + 'scripts/check-docker-deps-stage.mjs', + 'scripts/check-release-package-bootstrap.mjs', + 'scripts/release-package-map.mjs', + 'scripts/docker-onboard-smoke.sh', +]; + +export function scanCITampering(files) { + return files + .filter(f => f.filename.startsWith('.github/workflows/') && f.status !== 'removed') + .map(f => ({ check: 'ci-tampering', file: f.filename })); +} + +export function scanBuildScripts(files) { + return files + .filter(f => CI_BUILD_SCRIPTS.includes(f.filename) && f.status !== 'removed') + .map(f => ({ check: 'build-script-change', file: f.filename })); +} + +export function scanSupplyChain(files) { + const lockfile = files.find(f => f.filename === 'pnpm-lock.yaml'); + if (!lockfile?.patch) return []; + + const added = new Set(); + const removed = new Set(); + + for (const line of lockfile.patch.split('\n')) { + const entry = parseLockfilePackageDiffEntry(line); + if (!entry) continue; + if (entry.sign === '+') added.add(entry.packageName); + if (entry.sign === '-') removed.add(entry.packageName); + } + + const netNew = [...added].filter(p => !removed.has(p)); + return netNew.length ? [{ check: 'supply-chain', packages: netNew }] : []; +} + +function parseLockfilePackageDiffEntry(line) { + const match = line.match(/^([+-])\s*(.+?)\s*$/); + if (!match) return null; + + let [, sign, rawEntry] = match; + if (!rawEntry.endsWith(':')) return null; + + rawEntry = rawEntry.slice(0, -1).trim(); + if ((rawEntry.startsWith("'") && rawEntry.endsWith("'")) || (rawEntry.startsWith('"') && rawEntry.endsWith('"'))) { + rawEntry = rawEntry.slice(1, -1); + } + rawEntry = rawEntry.replace(/\(.*$/, '').trim(); + + const versionSep = rawEntry.lastIndexOf('@'); + if (versionSep <= 0 || versionSep === rawEntry.length - 1) return null; + + const packageName = rawEntry.slice(0, versionSep); + if (!/^(?:@[^/\s:]+\/)?[A-Za-z0-9._-][A-Za-z0-9._/-]*$/.test(packageName)) return null; + + return { sign, packageName }; +} + +const TEST_FILE_RE = /\.(test|spec)\.(ts|js|tsx|jsx)$|\/(?:__tests__|tests?)\//; +const SUSPICIOUS_PATTERNS = [ + { name: 'outbound-network', re: /\+.*(fetch\(|axios\.|http\.request|https\.request)/ }, + { name: 'env-var-read', re: /\+.*process\.env\.(?!(?:NODE_ENV|CI|TEST|VITEST|npm_))([A-Z_]{4,})/ }, + { name: 'shell-exec', re: /\+.*(execSync\(|spawnSync\(|exec\(|spawn\()/ }, + { name: 'absolute-file-read', re: /\+.*(readFile|readFileSync)\s*\(\s*["'`]?\// }, +]; + +export function scanTestPatterns(files) { + const flags = []; + for (const file of files) { + if (!TEST_FILE_RE.test(file.filename) || !file.patch) continue; + for (const { name, re } of SUSPICIOUS_PATTERNS) { + if (re.test(file.patch)) { + flags.push({ check: 'suspicious-test', file: file.filename, pattern: name }); + } + } + } + return flags; +} + +const SENSITIVE_PATHS = [ + // Advisory 1: codex-local adapter (inherited ChatGPT/Gmail OAuth scopes) + 'packages/adapters/codex-local/', + // Advisory 2 & 11: OS command injection / privilege escalation via provisionCommand / cleanupCommand + 'server/src/services/workspace-realization.ts', + 'server/src/routes/execution-workspaces.ts', + 'server/src/routes/workspace-command-authz.ts', + // Advisory 3 & 6: Cross-tenant agent API key minting and IDOR on /agents/:id/keys + 'server/src/routes/agents.ts', + // Advisory 4: Approval decision attribution spoofing via decidedByUserId + 'server/src/routes/approvals.ts', + // Advisory 5: Stored XSS via javascript: URLs in MarkdownBody (urlTransform) + 'ui/src/components/MarkdownBody.tsx', + // Advisory 7: Unauthenticated access to authenticated-mode endpoints + 'server/src/routes/authz.ts', + // Advisory 8: Unauthenticated RCE via import authorization bypass + 'server/src/routes/companies.ts', + // Advisory 9: Malicious skills able to exfiltrate / destroy user data + 'server/src/routes/company-skills.ts', + // Advisory 10: Arbitrary file read via agent-controlled instructionsFilePath + 'server/src/services/agent-instructions.ts', +]; + +export function scanSensitivePaths(files) { + return files + .filter(f => f.status !== 'removed' && SENSITIVE_PATHS.some(p => f.filename.startsWith(p))) + .map(f => ({ + check: 'sensitive-path', + file: f.filename, + advisoryPath: SENSITIVE_PATHS.find(p => f.filename.startsWith(p)), + })); +} + +function buildContentsPath(repo, filename, ref) { + return `/repos/${repo}/contents/${filename}?${new URLSearchParams({ ref }).toString()}`; +} + +export async function validateSensitivePaths(token, repo, prNumber, baseRef, fetchFromGitHub = ghFetch) { + const resolvedBaseRef = await resolveBaseRef(fetchFromGitHub, token, repo, prNumber, baseRef); + const stale = []; + await Promise.all(SENSITIVE_PATHS.map(async (path) => { + try { + await fetchFromGitHub(buildContentsPath(repo, path, resolvedBaseRef), token); + } catch (err) { + // 404 means the file/directory no longer exists at this path + if (String(err.message).includes('404')) stale.push(path); + // Other errors (network, rate limit) — re-throw so we don't silently miss them + else throw err; + } + })); + return stale; +} + +// ── Advisory creation ───────────────────────────────────────────────────────── + +const SEVERITY_MAP = { + 'supply-chain': 'critical', + 'sensitive-path': 'critical', + 'secret-scan': 'high', + 'ci-tampering': 'high', + 'suspicious-test': 'high', + 'build-script-change': 'medium', +}; + +const SEVERITY_ORDER = ['low', 'medium', 'high', 'critical']; + +function worstSeverity(flags) { + return flags.reduce((worst, f) => { + const s = SEVERITY_MAP[f.check] ?? 'medium'; + return SEVERITY_ORDER.indexOf(s) > SEVERITY_ORDER.indexOf(worst) ? s : worst; + }, 'low'); +} + +export function buildAdvisoryPayload(prNumber, prTitle, flags) { + const checkNames = [...new Set(flags.map(f => f.check))].join(', '); + return { + summary: `🚨 Security flag — PR #${prNumber}: ${checkNames}`, + description: [ + `**PR:** #${prNumber} — ${prTitle}`, + `**Checks triggered:** ${checkNames}`, + '', + '**Details:**', + ...flags.map(f => [ + `- \`${f.check}\`: ${f.file ?? ''}`, + f.pattern ? ` (pattern: ${f.pattern})` : '', + f.packages ? ` (packages: ${f.packages.join(', ')})` : '', + f.line ? `\n \`${f.line}\`` : '', + ].join('')), + '', + '> This advisory was created automatically by commitperclip. Review and dismiss if not a real concern.', + ].join('\n'), + severity: worstSeverity(flags), + vulnerabilities: [], + }; +} + +export async function syncDraftAdvisory(fetchImpl, token, repo, prNumber, prTitle, flags) { + const existing = await findExistingDraftAdvisory(fetchImpl, token, repo, prNumber); + const payload = buildAdvisoryPayload(prNumber, prTitle, flags); + + if (existing) { + const advisoryId = existing.ghsa_id ?? existing.id; + if (!advisoryId) { + throw new Error(`Existing advisory for PR #${prNumber} is missing both ghsa_id and id.`); + } + + return fetchImpl(`/repos/${repo}/security-advisories/${advisoryId}`, token, { + method: 'PATCH', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify(payload), + }); + } + + return fetchImpl(`/repos/${repo}/security-advisories`, token, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify(payload), + }); +} + +export async function findExistingDraftAdvisory(fetchImpl, token, repo, prNumber) { + const prMarker = `PR #${prNumber}`; + + for (let page = 1; ; page += 1) { + const advisories = await fetchImpl( + `/repos/${repo}/security-advisories?state=draft&per_page=100&page=${page}`, + token, + ); + + if (!Array.isArray(advisories) || advisories.length === 0) return null; + + const existing = advisories.find(advisory => + typeof advisory?.summary === 'string' && advisory.summary.includes(prMarker) + ); + if (existing) return existing; + + if (advisories.length < 100) return null; + } +} + +export async function postSecurityCheckRun(fetchImpl, token, repo, headSha, hasFlags) { + await fetchImpl(`/repos/${repo}/check-runs`, token, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify(hasFlags ? { + name: 'security-review', + head_sha: headSha, + status: 'in_progress', + output: { + title: 'Security Review Pending', + summary: 'This PR has been flagged for manual security review by a maintainer. No action needed from you.', + }, + } : { + name: 'security-review', + head_sha: headSha, + status: 'completed', + conclusion: 'success', + output: { + title: 'Security Review Passed', + summary: 'No security concerns detected.', + }, + }), + }); +} + +// ── Main ────────────────────────────────────────────────────────────────────── + +async function main() { + const { GH_TOKEN, GH_REPO, PR_NUMBER } = process.env; + + if (!GH_TOKEN || !GH_REPO || !PR_NUMBER) { + console.error('ERROR: GH_TOKEN, GH_REPO, PR_NUMBER required'); + process.exit(1); + } + + // Sanitize inputs before use in URL construction (prevents SSRF) + const prNumber = parseInt(PR_NUMBER, 10); + if (!Number.isInteger(prNumber) || prNumber <= 0) { + console.error('ERROR: PR_NUMBER must be a positive integer'); + process.exit(1); + } + if (!/^[a-zA-Z0-9_.-]+\/[a-zA-Z0-9_.-]+$/.test(GH_REPO)) { + console.error('ERROR: GH_REPO must be in owner/repo format'); + process.exit(1); + } + + // Validate SENSITIVE_PATHS — fails loudly if any have been refactored away on the PR base branch + const stalePaths = await validateSensitivePaths(GH_TOKEN, GH_REPO, prNumber); + if (stalePaths.length > 0) { + console.error('ERROR: Stale sensitive paths in check-pr-security.mjs:'); + for (const p of stalePaths) console.error(` - ${p}`); + console.error(''); + console.error('These paths no longer exist on the PR base branch. The security gate will silently produce no signal for them.'); + console.error('Update SENSITIVE_PATHS in check-pr-security.mjs to reflect the current code structure.'); + process.exit(1); + } + + const [pr, files] = await Promise.all([ + ghFetch(`/repos/${GH_REPO}/pulls/${prNumber}`, GH_TOKEN), + fetchAllPullRequestFiles(ghFetch, GH_REPO, prNumber, GH_TOKEN), + ]); + + const allFlags = [ + ...scanSecrets(files), + ...scanCITampering(files), + ...scanBuildScripts(files), + ...scanSupplyChain(files), + ...scanTestPatterns(files), + ...scanSensitivePaths(files), + ]; + + if (allFlags.length > 0) { + console.error(`[security] ${allFlags.length} flag(s) detected — creating draft advisory and pending check run`); + await Promise.all([ + syncDraftAdvisory(ghFetch, GH_TOKEN, GH_REPO, prNumber, pr.title, allFlags), + postSecurityCheckRun(ghFetch, GH_TOKEN, GH_REPO, pr.head.sha, true), + ]); + } else { + console.log('[security] all clear'); + await postSecurityCheckRun(ghFetch, GH_TOKEN, GH_REPO, pr.head.sha, false); + } + + // Always exit 0 — security flags are silent, never block the PR publicly + process.exit(0); +} + +if (process.argv[1] === fileURLToPath(import.meta.url)) { + main().catch(e => { console.error(e.message); process.exit(1); }); +} diff --git a/.github/scripts/check-pr-template.mjs b/.github/scripts/check-pr-template.mjs new file mode 100644 index 00000000..851ea5cd --- /dev/null +++ b/.github/scripts/check-pr-template.mjs @@ -0,0 +1,88 @@ +#!/usr/bin/env node +/** + * check-pr-template.mjs + * Checks that a PR body contains all required sections from the PR template. + * Export: checkTemplate(prBody: string) → { passed: boolean, failures: string[] } + */ +import { fileURLToPath } from 'node:url'; + +const REQUIRED_SECTIONS = [ + { heading: '## Thinking Path', minSentences: 3 }, + { heading: '## What Changed', minSentences: 1 }, + { heading: '## Verification', minSentences: 1 }, + { heading: '## Risks', minSentences: 1 }, + { heading: '## Model Used', minSentences: 1 }, +]; + +const MODEL_PLACEHOLDERS = [ + 'provider, model id', + 'your model', + '', +]; + +function extractSectionContent(body, heading) { + const idx = body.indexOf(heading); + if (idx === -1) return null; + const after = body.slice(idx + heading.length); + const nextHeading = after.search(/\n## /); + return (nextHeading === -1 ? after : after.slice(0, nextHeading)).trim(); +} + +function countSentences(text) { + // Split on terminal punctuation, bullet/quote line starts (`-`, `*`, `>`), or + // blank lines so non-prose Thinking Paths (bullet lists, blockquotes) are + // counted by item rather than as a single sentence. + return text.split(/[.!?]+\s+|\n\s*[-*>]+\s+|\n{2,}/).filter(s => s.trim().length > 5).length; +} + +export function checkTemplate(body) { + const failures = []; + + if (!body || !body.trim()) { + for (const { heading } of REQUIRED_SECTIONS) { + failures.push(`Missing section: **${heading}**`); + } + return { passed: false, failures }; + } + + for (const { heading, minSentences } of REQUIRED_SECTIONS) { + const content = extractSectionContent(body, heading); + + if (content === null) { + failures.push(`Missing section: **${heading}**`); + continue; + } + + if (!content || content === '_No response_' || /^