[codex] Add plugin orchestration host APIs (#4114)

## Thinking Path > - Paperclip orchestrates AI agents for zero-human companies. > - The plugin system is the extension path for optional capabilities that should not require core product changes for every integration. > - Plugins need scoped host APIs for issue orchestration, documents, wakeups, summaries, activity attribution, and isolated database state. > - Without those host APIs, richer plugins either cannot coordinate Paperclip work safely or need privileged core-side special cases. > - This pull request adds the plugin orchestration host surface, scoped route dispatch, a database namespace layer, and a smoke plugin that exercises the contract. > - The benefit is a broader plugin API that remains company-scoped, auditable, and covered by tests. ## What Changed - Added plugin orchestration host APIs for issue creation, document access, wakeups, summaries, plugin-origin activity, and scoped API route dispatch. - Added plugin database namespace tables, schema exports, migration checks, and idempotent replay coverage under migration `0059_plugin_database_namespaces`. - Added shared plugin route/API types and validators used by server and SDK boundaries. - Expanded plugin SDK types, protocol helpers, worker RPC host behavior, and testing utilities for orchestration flows. - Added the `plugin-orchestration-smoke-example` package to exercise scoped routes, restricted database namespaces, issue orchestration, documents, wakeups, summaries, and UI status surfaces. - Kept the new orchestration smoke fixture out of the root pnpm workspace importer so this PR preserves the repository policy of not committing `pnpm-lock.yaml`. - Updated plugin docs and database docs for the new orchestration and database namespace surfaces. - Rebased the branch onto `public-gh/master`, resolved conflicts, and removed `pnpm-lock.yaml` from the final PR diff. ## Verification - `pnpm install --frozen-lockfile` - `pnpm --filter @paperclipai/db typecheck` - `pnpm exec vitest run packages/db/src/client.test.ts` - `pnpm exec vitest run server/src/__tests__/plugin-database.test.ts server/src/__tests__/plugin-orchestration-apis.test.ts server/src/__tests__/plugin-routes-authz.test.ts server/src/__tests__/plugin-scoped-api-routes.test.ts server/src/__tests__/plugin-sdk-orchestration-contract.test.ts` - From `packages/plugins/examples/plugin-orchestration-smoke-example`: `pnpm exec vitest run --config ./vitest.config.ts` - `pnpm --dir packages/plugins/examples/plugin-orchestration-smoke-example run typecheck` - `pnpm --filter @paperclipai/server typecheck` - PR CI on latest head `293fc67c`: `policy`, `verify`, `e2e`, and `security/snyk` all passed. ## Risks - Medium risk: this expands plugin host authority, so route auth, company scoping, and plugin-origin activity attribution need careful review. - Medium risk: database namespace migration behavior must remain idempotent for environments that may have seen earlier branch versions. - Medium risk: the orchestration smoke fixture is intentionally excluded from the root workspace importer to avoid a `pnpm-lock.yaml` PR diff; direct fixture verification remains listed above. - Low operational risk from the PR setup itself: the branch is rebased onto current `master`, the migration is ordered after upstream `0057`/`0058`, and `pnpm-lock.yaml` is not in the final diff. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. Roadmap checked: this work aligns with the completed Plugin system milestone and extends the plugin surface rather than duplicating an unrelated planned core feature. ## Model Used - OpenAI Codex, GPT-5-based coding agent in a tool-enabled CLI environment. Exact hosted model build and context-window size are not exposed by the runtime; reasoning/tool use were enabled for repository inspection, editing, testing, git operations, and PR creation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] If this change affects the UI, I have included before/after screenshots (N/A: no core UI screen change; example plugin UI contract is covered by tests) - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-06-14 18:10:39 +09:00 · 2026-04-20 08:52:51 -05:00 · 2026-04-20 08:52:51 -05:00 · 9c6f551595
commit 9c6f551595
parent 16b2b84d84
53 changed files with 5584 additions and 53 deletions
--- a/server/src/services/plugin-database.ts
+++ b/server/src/services/plugin-database.ts
@ -0,0 +1,498 @@
+import { createHash } from "node:crypto";
+import { readdir, readFile } from "node:fs/promises";
+import path from "node:path";
+import { and, eq, sql } from "drizzle-orm";
+import type { SQL } from "drizzle-orm";
+import type { Db } from "@paperclipai/db";
+import {
+  pluginDatabaseNamespaces,
+  pluginMigrations,
+  plugins,
+} from "@paperclipai/db";
+import type {
+  PaperclipPluginManifestV1,
+  PluginDatabaseCoreReadTable,
+  PluginMigrationRecord,
+} from "@paperclipai/shared";
+
+const IDENTIFIER_RE = /^[A-Za-z_][A-Za-z0-9_]*$/;
+const MAX_POSTGRES_IDENTIFIER_LENGTH = 63;
+
+type SqlRef = { schema: string; table: string; keyword: string };
+
+export type PluginDatabaseRuntimeResult<T = Record<string, unknown>> = {
+  rows?: T[];
+  rowCount?: number;
+};
+
+export function derivePluginDatabaseNamespace(
+  pluginKey: string,
+  namespaceSlug?: string,
+): string {
+  const hash = createHash("sha256").update(pluginKey).digest("hex").slice(0, 10);
+  const slug = (namespaceSlug ?? pluginKey)
+    .toLowerCase()
+    .replace(/[^a-z0-9_]+/g, "_")
+    .replace(/^_+|_+$/g, "")
+    .replace(/_+/g, "_")
+    .slice(0, 36) || "plugin";
+  const namespace = `plugin_${slug}_${hash}`;
+  return namespace.slice(0, MAX_POSTGRES_IDENTIFIER_LENGTH);
+}
+
+function assertIdentifier(value: string, label = "identifier"): string {
+  if (!IDENTIFIER_RE.test(value)) {
+    throw new Error(`Unsafe SQL ${label}: ${value}`);
+  }
+  return value;
+}
+
+function quoteIdentifier(value: string): string {
+  return `"${assertIdentifier(value).replaceAll("\"", "\"\"")}"`;
+}
+
+function splitSqlStatements(input: string): string[] {
+  const statements: string[] = [];
+  let start = 0;
+  let quote: "'" | "\"" | null = null;
+  let lineComment = false;
+  let blockComment = false;
+
+  for (let i = 0; i < input.length; i += 1) {
+    const char = input[i]!;
+    const next = input[i + 1];
+
+    if (lineComment) {
+      if (char === "\n") lineComment = false;
+      continue;
+    }
+    if (blockComment) {
+      if (char === "*" && next === "/") {
+        blockComment = false;
+        i += 1;
+      }
+      continue;
+    }
+    if (quote) {
+      if (char === quote) {
+        if (next === quote) {
+          i += 1;
+        } else {
+          quote = null;
+        }
+      }
+      continue;
+    }
+    if (char === "-" && next === "-") {
+      lineComment = true;
+      i += 1;
+      continue;
+    }
+    if (char === "/" && next === "*") {
+      blockComment = true;
+      i += 1;
+      continue;
+    }
+    if (char === "'" || char === "\"") {
+      quote = char;
+      continue;
+    }
+    if (char === ";") {
+      const statement = input.slice(start, i).trim();
+      if (statement) statements.push(statement);
+      start = i + 1;
+    }
+  }
+
+  const trailing = input.slice(start).trim();
+  if (trailing) statements.push(trailing);
+  return statements;
+}
+
+function stripSqlForKeywordScan(input: string): string {
+  return input
+    .replace(/'([^']|'')*'/g, "''")
+    .replace(/"([^"]|"")*"/g, "\"\"")
+    .replace(/--.*$/gm, "")
+    .replace(/\/\*[\s\S]*?\*\//g, "");
+}
+
+function normaliseSql(input: string): string {
+  return stripSqlForKeywordScan(input).replace(/\s+/g, " ").trim().toLowerCase();
+}
+
+function extractQualifiedRefs(statement: string): SqlRef[] {
+  const refs: SqlRef[] = [];
+  const patterns = [
+    /\b(from|join|references|into|update)\s+"?([A-Za-z_][A-Za-z0-9_]*)"?\."?([A-Za-z_][A-Za-z0-9_]*)"?/gi,
+    /\b(alter\s+table|create\s+table|create\s+view|drop\s+table|truncate\s+table)\s+(?:if\s+(?:not\s+)?exists\s+)?"?([A-Za-z_][A-Za-z0-9_]*)"?\."?([A-Za-z_][A-Za-z0-9_]*)"?/gi,
+  ];
+
+  for (const pattern of patterns) {
+    for (const match of statement.matchAll(pattern)) {
+      refs.push({ keyword: match[1]!.toLowerCase(), schema: match[2]!, table: match[3]! });
+    }
+  }
+  return refs;
+}
+
+function assertAllowedPublicRead(
+  ref: SqlRef,
+  allowedCoreReadTables: ReadonlySet<string>,
+): void {
+  if (ref.schema !== "public") return;
+  if (!allowedCoreReadTables.has(ref.table)) {
+    throw new Error(`Plugin SQL references public.${ref.table}, which is not whitelisted`);
+  }
+  if (!["from", "join", "references"].includes(ref.keyword)) {
+    throw new Error(`Plugin SQL cannot mutate or define objects in public.${ref.table}`);
+  }
+}
+
+function assertNoBannedSql(statement: string): void {
+  const normalized = normaliseSql(statement);
+  const banned = [
+    /\bcreate\s+extension\b/,
+    /\bcreate\s+(?:event\s+)?trigger\b/,
+    /\bcreate\s+(?:or\s+replace\s+)?function\b/,
+    /\bcreate\s+language\b/,
+    /\bgrant\b/,
+    /\brevoke\b/,
+    /\bsecurity\s+definer\b/,
+    /\bcopy\b/,
+    /\bcall\b/,
+    /\bdo\s+(?:\$\$|language\b)/,
+  ];
+  const matched = banned.find((pattern) => pattern.test(normalized));
+  if (matched) {
+    throw new Error(`Plugin SQL contains a disallowed statement or clause: ${matched.source}`);
+  }
+}
+
+export function validatePluginMigrationStatement(
+  statement: string,
+  namespace: string,
+  coreReadTables: readonly PluginDatabaseCoreReadTable[] = [],
+): void {
+  assertIdentifier(namespace, "namespace");
+  assertNoBannedSql(statement);
+
+  const normalized = normaliseSql(statement);
+  if (/^\s*(drop|truncate)\b/.test(normalized)) {
+    throw new Error("Destructive plugin migrations are not allowed in Phase 1");
+  }
+
+  const ddlAllowed = /^(create|alter|comment)\b/.test(normalized);
+  if (!ddlAllowed) {
+    throw new Error("Plugin migrations may contain DDL statements only");
+  }
+
+  const refs = extractQualifiedRefs(statement);
+  if (refs.length === 0 && !normalized.startsWith("comment ")) {
+    throw new Error("Plugin migration objects must use fully qualified schema names");
+  }
+
+  const allowedCoreReadTables = new Set(coreReadTables);
+  for (const ref of refs) {
+    if (ref.schema === namespace) continue;
+    if (ref.schema === "public") {
+      assertAllowedPublicRead(ref, allowedCoreReadTables);
+      continue;
+    }
+    throw new Error(`Plugin SQL references schema "${ref.schema}" outside namespace "${namespace}"`);
+  }
+}
+
+export function validatePluginRuntimeQuery(
+  query: string,
+  namespace: string,
+  coreReadTables: readonly PluginDatabaseCoreReadTable[] = [],
+): void {
+  const statements = splitSqlStatements(query);
+  if (statements.length !== 1) {
+    throw new Error("Plugin runtime SQL must contain exactly one statement");
+  }
+  const statement = statements[0]!;
+  assertNoBannedSql(statement);
+  const normalized = normaliseSql(statement);
+  if (!normalized.startsWith("select ") && !normalized.startsWith("with ")) {
+    throw new Error("ctx.db.query only allows SELECT statements");
+  }
+  if (/\b(insert|update|delete|alter|create|drop|truncate)\b/.test(normalized)) {
+    throw new Error("ctx.db.query cannot contain mutation or DDL keywords");
+  }
+
+  const allowedCoreReadTables = new Set(coreReadTables);
+  for (const ref of extractQualifiedRefs(statement)) {
+    if (ref.schema === namespace) continue;
+    if (ref.schema === "public") {
+      assertAllowedPublicRead(ref, allowedCoreReadTables);
+      continue;
+    }
+    throw new Error(`ctx.db.query cannot read schema "${ref.schema}"`);
+  }
+}
+
+export function validatePluginRuntimeExecute(query: string, namespace: string): void {
+  const statements = splitSqlStatements(query);
+  if (statements.length !== 1) {
+    throw new Error("Plugin runtime SQL must contain exactly one statement");
+  }
+  const statement = statements[0]!;
+  assertNoBannedSql(statement);
+  const normalized = normaliseSql(statement);
+  if (!/^(insert\s+into|update|delete\s+from)\b/.test(normalized)) {
+    throw new Error("ctx.db.execute only allows INSERT, UPDATE, or DELETE");
+  }
+  if (/\b(alter|create|drop|truncate)\b/.test(normalized)) {
+    throw new Error("ctx.db.execute cannot contain DDL keywords");
+  }
+
+  const refs = extractQualifiedRefs(statement);
+  const target = refs.find((ref) => ["into", "update", "from"].includes(ref.keyword));
+  if (!target || target.schema !== namespace) {
+    throw new Error(`ctx.db.execute target must be inside plugin namespace "${namespace}"`);
+  }
+  for (const ref of refs) {
+    if (ref.schema !== namespace) {
+      throw new Error("ctx.db.execute cannot reference public or other non-plugin schemas");
+    }
+  }
+}
+
+function bindSql(statement: string, params: readonly unknown[] = []): SQL {
+  // Safe only after callers run the plugin SQL validators above.
+  if (params.length === 0) return sql.raw(statement);
+  const chunks: SQL[] = [];
+  let cursor = 0;
+  const placeholderPattern = /\$(\d+)/g;
+  const seen = new Set<number>();
+
+  for (const match of statement.matchAll(placeholderPattern)) {
+    const index = Number(match[1]);
+    if (!Number.isInteger(index) || index < 1 || index > params.length) {
+      throw new Error(`SQL placeholder $${match[1]} has no matching parameter`);
+    }
+    chunks.push(sql.raw(statement.slice(cursor, match.index)));
+    chunks.push(sql`${params[index - 1]}`);
+    seen.add(index);
+    cursor = match.index! + match[0].length;
+  }
+  chunks.push(sql.raw(statement.slice(cursor)));
+  if (seen.size !== params.length) {
+    throw new Error("Every ctx.db parameter must be referenced by a $n placeholder");
+  }
+  return sql.join(chunks, sql.raw(""));
+}
+
+async function listSqlMigrationFiles(migrationsDir: string): Promise<string[]> {
+  const entries = await readdir(migrationsDir, { withFileTypes: true });
+  return entries
+    .filter((entry) => entry.isFile() && entry.name.endsWith(".sql"))
+    .map((entry) => entry.name)
+    .sort((a, b) => a.localeCompare(b));
+}
+
+function resolveMigrationsDir(packageRoot: string, migrationsDir: string): string {
+  const resolvedRoot = path.resolve(packageRoot);
+  const resolvedDir = path.resolve(resolvedRoot, migrationsDir);
+  const relative = path.relative(resolvedRoot, resolvedDir);
+  if (relative.startsWith("..") || path.isAbsolute(relative)) {
+    throw new Error(`Plugin migrationsDir escapes package root: ${migrationsDir}`);
+  }
+  return resolvedDir;
+}
+
+export function pluginDatabaseService(db: Db) {
+  async function getPluginRecord(pluginId: string) {
+    const rows = await db.select().from(plugins).where(eq(plugins.id, pluginId)).limit(1);
+    const plugin = rows[0];
+    if (!plugin) throw new Error(`Plugin not found: ${pluginId}`);
+    return plugin;
+  }
+
+  async function ensureNamespace(pluginId: string, manifest: PaperclipPluginManifestV1) {
+    if (!manifest.database) return null;
+    const namespaceName = derivePluginDatabaseNamespace(
+      manifest.id,
+      manifest.database.namespaceSlug,
+    );
+    await db.execute(sql.raw(`CREATE SCHEMA IF NOT EXISTS ${quoteIdentifier(namespaceName)}`));
+    const rows = await db
+      .insert(pluginDatabaseNamespaces)
+      .values({
+        pluginId,
+        pluginKey: manifest.id,
+        namespaceName,
+        namespaceMode: "schema",
+        status: "active",
+      })
+      .onConflictDoUpdate({
+        target: pluginDatabaseNamespaces.pluginId,
+        set: {
+          pluginKey: manifest.id,
+          namespaceName,
+          namespaceMode: "schema",
+          status: "active",
+          updatedAt: new Date(),
+        },
+      })
+      .returning();
+    return rows[0] ?? null;
+  }
+
+  async function getNamespace(pluginId: string) {
+    const rows = await db
+      .select()
+      .from(pluginDatabaseNamespaces)
+      .where(eq(pluginDatabaseNamespaces.pluginId, pluginId))
+      .limit(1);
+    return rows[0] ?? null;
+  }
+
+  async function getRuntimeNamespace(pluginId: string) {
+    const namespace = await getNamespace(pluginId);
+    if (!namespace || namespace.status !== "active") {
+      throw new Error("Plugin database namespace is not active");
+    }
+    return namespace.namespaceName;
+  }
+
+  async function recordMigrationFailure(input: {
+    pluginId: string;
+    pluginKey: string;
+    namespaceName: string;
+    migrationKey: string;
+    checksum: string;
+    pluginVersion: string;
+    error: unknown;
+  }): Promise<void> {
+    const message = input.error instanceof Error ? input.error.message : String(input.error);
+    await db
+      .insert(pluginMigrations)
+      .values({
+        pluginId: input.pluginId,
+        pluginKey: input.pluginKey,
+        namespaceName: input.namespaceName,
+        migrationKey: input.migrationKey,
+        checksum: input.checksum,
+        pluginVersion: input.pluginVersion,
+        status: "failed",
+        errorMessage: message,
+      })
+      .onConflictDoUpdate({
+        target: [pluginMigrations.pluginId, pluginMigrations.migrationKey],
+        set: {
+          checksum: input.checksum,
+          pluginVersion: input.pluginVersion,
+          status: "failed",
+          errorMessage: message,
+          startedAt: new Date(),
+          appliedAt: null,
+        },
+      });
+    await db
+      .update(pluginDatabaseNamespaces)
+      .set({ status: "migration_failed", updatedAt: new Date() })
+      .where(eq(pluginDatabaseNamespaces.pluginId, input.pluginId));
+  }
+
+  return {
+    ensureNamespace,
+
+    async applyMigrations(pluginId: string, manifest: PaperclipPluginManifestV1, packageRoot: string) {
+      if (!manifest.database) return null;
+      const namespace = await ensureNamespace(pluginId, manifest);
+      if (!namespace) return null;
+
+      const migrationDir = resolveMigrationsDir(packageRoot, manifest.database.migrationsDir);
+      const migrationFiles = await listSqlMigrationFiles(migrationDir);
+      const coreReadTables = manifest.database.coreReadTables ?? [];
+      const lockKey = Number.parseInt(createHash("sha256").update(pluginId).digest("hex").slice(0, 12), 16);
+
+      await db.transaction(async (tx) => {
+        await tx.execute(sql`SELECT pg_advisory_xact_lock(${lockKey})`);
+        for (const migrationKey of migrationFiles) {
+          const content = await readFile(path.join(migrationDir, migrationKey), "utf8");
+          const checksum = createHash("sha256").update(content).digest("hex");
+          const existingRows = await tx
+            .select()
+            .from(pluginMigrations)
+            .where(and(eq(pluginMigrations.pluginId, pluginId), eq(pluginMigrations.migrationKey, migrationKey)))
+            .limit(1);
+          const existing = existingRows[0] as PluginMigrationRecord | undefined;
+          if (existing?.status === "applied") {
+            if (existing.checksum !== checksum) {
+              throw new Error(`Plugin migration checksum mismatch for ${migrationKey}`);
+            }
+            continue;
+          }
+
+          const statements = splitSqlStatements(content);
+          try {
+            if (statements.length === 0) {
+              throw new Error(`Plugin migration ${migrationKey} is empty`);
+            }
+            for (const statement of statements) {
+              validatePluginMigrationStatement(statement, namespace.namespaceName, coreReadTables);
+              await tx.execute(sql.raw(statement));
+            }
+            await tx
+              .insert(pluginMigrations)
+              .values({
+                pluginId,
+                pluginKey: manifest.id,
+                namespaceName: namespace.namespaceName,
+                migrationKey,
+                checksum,
+                pluginVersion: manifest.version,
+                status: "applied",
+                appliedAt: new Date(),
+              })
+              .onConflictDoUpdate({
+                target: [pluginMigrations.pluginId, pluginMigrations.migrationKey],
+                set: {
+                  checksum,
+                  pluginVersion: manifest.version,
+                  status: "applied",
+                  errorMessage: null,
+                  startedAt: new Date(),
+                  appliedAt: new Date(),
+                },
+              });
+          } catch (error) {
+            await recordMigrationFailure({
+              pluginId,
+              pluginKey: manifest.id,
+              namespaceName: namespace.namespaceName,
+              migrationKey,
+              checksum,
+              pluginVersion: manifest.version,
+              error,
+            });
+            throw error;
+          }
+        }
+      });
+
+      return namespace;
+    },
+
+    getRuntimeNamespace,
+
+    async query<T = Record<string, unknown>>(pluginId: string, statement: string, params?: unknown[]): Promise<T[]> {
+      const plugin = await getPluginRecord(pluginId);
+      const namespace = await getRuntimeNamespace(pluginId);
+      validatePluginRuntimeQuery(statement, namespace, plugin.manifestJson.database?.coreReadTables ?? []);
+      const result = await db.execute(bindSql(statement, params));
+      return Array.from(result as Iterable<T>);
+    },
+
+    async execute(pluginId: string, statement: string, params?: unknown[]): Promise<{ rowCount: number }> {
+      const namespace = await getRuntimeNamespace(pluginId);
+      validatePluginRuntimeExecute(statement, namespace);
+      const result = await db.execute(bindSql(statement, params));
+      return { rowCount: Number((result as { count?: number | string }).count ?? 0) };
+    },
+  };
+}