[codex] Add plugin orchestration host APIs (#4114)

## Thinking Path

> - Paperclip orchestrates AI agents for zero-human companies.
> - The plugin system is the extension path for optional capabilities
that should not require core product changes for every integration.
> - Plugins need scoped host APIs for issue orchestration, documents,
wakeups, summaries, activity attribution, and isolated database state.
> - Without those host APIs, richer plugins either cannot coordinate
Paperclip work safely or need privileged core-side special cases.
> - This pull request adds the plugin orchestration host surface, scoped
route dispatch, a database namespace layer, and a smoke plugin that
exercises the contract.
> - The benefit is a broader plugin API that remains company-scoped,
auditable, and covered by tests.

## What Changed

- Added plugin orchestration host APIs for issue creation, document
access, wakeups, summaries, plugin-origin activity, and scoped API route
dispatch.
- Added plugin database namespace tables, schema exports, migration
checks, and idempotent replay coverage under migration
`0059_plugin_database_namespaces`.
- Added shared plugin route/API types and validators used by server and
SDK boundaries.
- Expanded plugin SDK types, protocol helpers, worker RPC host behavior,
and testing utilities for orchestration flows.
- Added the `plugin-orchestration-smoke-example` package to exercise
scoped routes, restricted database namespaces, issue orchestration,
documents, wakeups, summaries, and UI status surfaces.
- Kept the new orchestration smoke fixture out of the root pnpm
workspace importer so this PR preserves the repository policy of not
committing `pnpm-lock.yaml`.
- Updated plugin docs and database docs for the new orchestration and
database namespace surfaces.
- Rebased the branch onto `public-gh/master`, resolved conflicts, and
removed `pnpm-lock.yaml` from the final PR diff.

## Verification

- `pnpm install --frozen-lockfile`
- `pnpm --filter @paperclipai/db typecheck`
- `pnpm exec vitest run packages/db/src/client.test.ts`
- `pnpm exec vitest run server/src/__tests__/plugin-database.test.ts
server/src/__tests__/plugin-orchestration-apis.test.ts
server/src/__tests__/plugin-routes-authz.test.ts
server/src/__tests__/plugin-scoped-api-routes.test.ts
server/src/__tests__/plugin-sdk-orchestration-contract.test.ts`
- From `packages/plugins/examples/plugin-orchestration-smoke-example`:
`pnpm exec vitest run --config ./vitest.config.ts`
- `pnpm --dir
packages/plugins/examples/plugin-orchestration-smoke-example run
typecheck`
- `pnpm --filter @paperclipai/server typecheck`
- PR CI on latest head `293fc67c`: `policy`, `verify`, `e2e`, and
`security/snyk` all passed.

## Risks

- Medium risk: this expands plugin host authority, so route auth,
company scoping, and plugin-origin activity attribution need careful
review.
- Medium risk: database namespace migration behavior must remain
idempotent for environments that may have seen earlier branch versions.
- Medium risk: the orchestration smoke fixture is intentionally excluded
from the root workspace importer to avoid a `pnpm-lock.yaml` PR diff;
direct fixture verification remains listed above.
- Low operational risk from the PR setup itself: the branch is rebased
onto current `master`, the migration is ordered after upstream
`0057`/`0058`, and `pnpm-lock.yaml` is not in the final diff.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

Roadmap checked: this work aligns with the completed Plugin system
milestone and extends the plugin surface rather than duplicating an
unrelated planned core feature.

## Model Used

- OpenAI Codex, GPT-5-based coding agent in a tool-enabled CLI
environment. Exact hosted model build and context-window size are not
exposed by the runtime; reasoning/tool use were enabled for repository
inspection, editing, testing, git operations, and PR creation.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] If this change affects the UI, I have included before/after
screenshots (N/A: no core UI screen change; example plugin UI contract
is covered by tests)
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Dotta 2026-04-20 08:52:51 -05:00 committed by GitHub
parent 16b2b84d84
commit 9c6f551595
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
53 changed files with 5584 additions and 53 deletions

View file

@ -138,7 +138,9 @@ export const ISSUE_PRIORITIES = ["critical", "high", "medium", "low"] as const;
export type IssuePriority = (typeof ISSUE_PRIORITIES)[number];
export const ISSUE_ORIGIN_KINDS = ["manual", "routine_execution"] as const;
export type IssueOriginKind = (typeof ISSUE_ORIGIN_KINDS)[number];
export type BuiltInIssueOriginKind = (typeof ISSUE_ORIGIN_KINDS)[number];
export type PluginIssueOriginKind = `plugin:${string}`;
export type IssueOriginKind = BuiltInIssueOriginKind | PluginIssueOriginKind;
export const ISSUE_RELATION_TYPES = ["blocks"] as const;
export type IssueRelationType = (typeof ISSUE_RELATION_TYPES)[number];
@ -498,6 +500,8 @@ export const PLUGIN_CAPABILITIES = [
"projects.read",
"project.workspaces.read",
"issues.read",
"issue.relations.read",
"issue.subtree.read",
"issue.comments.read",
"issue.documents.read",
"agents.read",
@ -506,9 +510,14 @@ export const PLUGIN_CAPABILITIES = [
"goals.update",
"activity.read",
"costs.read",
"issues.orchestration.read",
"database.namespace.read",
// Data Write
"issues.create",
"issues.update",
"issue.relations.write",
"issues.checkout",
"issues.wakeup",
"issue.comments.create",
"issue.documents.write",
"agents.pause",
@ -521,6 +530,8 @@ export const PLUGIN_CAPABILITIES = [
"activity.log.write",
"metrics.write",
"telemetry.track",
"database.namespace.migrate",
"database.namespace.write",
// Plugin State
"plugin.state.read",
"plugin.state.write",
@ -529,6 +540,7 @@ export const PLUGIN_CAPABILITIES = [
"events.emit",
"jobs.schedule",
"webhooks.receive",
"api.routes.register",
"http.outbound",
"secrets.read-ref",
// Agent Tools
@ -544,6 +556,51 @@ export const PLUGIN_CAPABILITIES = [
] as const;
export type PluginCapability = (typeof PLUGIN_CAPABILITIES)[number];
export const PLUGIN_DATABASE_NAMESPACE_MODES = ["schema"] as const;
export type PluginDatabaseNamespaceMode = (typeof PLUGIN_DATABASE_NAMESPACE_MODES)[number];
export const PLUGIN_DATABASE_NAMESPACE_STATUSES = [
"active",
"migration_failed",
] as const;
export type PluginDatabaseNamespaceStatus = (typeof PLUGIN_DATABASE_NAMESPACE_STATUSES)[number];
export const PLUGIN_DATABASE_MIGRATION_STATUSES = [
"applied",
"failed",
] as const;
export type PluginDatabaseMigrationStatus = (typeof PLUGIN_DATABASE_MIGRATION_STATUSES)[number];
export const PLUGIN_DATABASE_CORE_READ_TABLES = [
"companies",
"projects",
"goals",
"agents",
"issues",
"issue_documents",
"issue_relations",
"issue_comments",
"heartbeat_runs",
"cost_events",
"approvals",
"issue_approvals",
"budget_incidents",
] as const;
export type PluginDatabaseCoreReadTable = (typeof PLUGIN_DATABASE_CORE_READ_TABLES)[number];
export const PLUGIN_API_ROUTE_METHODS = ["GET", "POST", "PATCH", "DELETE"] as const;
export type PluginApiRouteMethod = (typeof PLUGIN_API_ROUTE_METHODS)[number];
export const PLUGIN_API_ROUTE_AUTH_MODES = ["board", "agent", "board-or-agent", "webhook"] as const;
export type PluginApiRouteAuthMode = (typeof PLUGIN_API_ROUTE_AUTH_MODES)[number];
export const PLUGIN_API_ROUTE_CHECKOUT_POLICIES = [
"none",
"required-for-agent-in-progress",
"always-for-agent",
] as const;
export type PluginApiRouteCheckoutPolicy = (typeof PLUGIN_API_ROUTE_CHECKOUT_POLICIES)[number];
/**
* UI extension slot types. Each slot type corresponds to a mount point in the
* Paperclip UI where plugin components can be rendered.
@ -742,6 +799,13 @@ export const PLUGIN_EVENT_TYPES = [
"issue.created",
"issue.updated",
"issue.comment.created",
"issue.document.created",
"issue.document.updated",
"issue.document.deleted",
"issue.relations.updated",
"issue.checked_out",
"issue.released",
"issue.assignment_wakeup_requested",
"agent.created",
"agent.updated",
"agent.status_changed",
@ -753,6 +817,8 @@ export const PLUGIN_EVENT_TYPES = [
"goal.updated",
"approval.created",
"approval.decided",
"budget.incident.opened",
"budget.incident.resolved",
"cost_event.created",
"activity.logged",
] as const;

View file

@ -83,6 +83,13 @@ export {
PLUGIN_JOB_RUN_STATUSES,
PLUGIN_JOB_RUN_TRIGGERS,
PLUGIN_WEBHOOK_DELIVERY_STATUSES,
PLUGIN_DATABASE_NAMESPACE_MODES,
PLUGIN_DATABASE_NAMESPACE_STATUSES,
PLUGIN_DATABASE_MIGRATION_STATUSES,
PLUGIN_DATABASE_CORE_READ_TABLES,
PLUGIN_API_ROUTE_METHODS,
PLUGIN_API_ROUTE_AUTH_MODES,
PLUGIN_API_ROUTE_CHECKOUT_POLICIES,
PLUGIN_EVENT_TYPES,
PLUGIN_BRIDGE_ERROR_CODES,
type CompanyStatus,
@ -96,6 +103,8 @@ export {
type AgentIconName,
type IssueStatus,
type IssuePriority,
type BuiltInIssueOriginKind,
type PluginIssueOriginKind,
type IssueOriginKind,
type IssueRelationType,
type SystemIssueDocumentKey,
@ -159,6 +168,13 @@ export {
type PluginJobRunStatus,
type PluginJobRunTrigger,
type PluginWebhookDeliveryStatus,
type PluginDatabaseNamespaceMode,
type PluginDatabaseNamespaceStatus,
type PluginDatabaseMigrationStatus,
type PluginDatabaseCoreReadTable,
type PluginApiRouteMethod,
type PluginApiRouteAuthMode,
type PluginApiRouteCheckoutPolicy,
type PluginEventType,
type PluginBridgeErrorCode,
} from "./constants.js";
@ -397,8 +413,13 @@ export type {
PluginLauncherDeclaration,
PluginMinimumHostVersion,
PluginUiDeclaration,
PluginDatabaseDeclaration,
PluginApiRouteCompanyResolution,
PluginApiRouteDeclaration,
PaperclipPluginManifestV1,
PluginRecord,
PluginDatabaseNamespaceRecord,
PluginMigrationRecord,
PluginStateRecord,
PluginConfig,
PluginEntityRecord,
@ -677,6 +698,8 @@ export {
pluginLauncherActionDeclarationSchema,
pluginLauncherRenderDeclarationSchema,
pluginLauncherDeclarationSchema,
pluginDatabaseDeclarationSchema,
pluginApiRouteDeclarationSchema,
pluginManifestV1Schema,
installPluginSchema,
upsertPluginConfigSchema,
@ -693,6 +716,8 @@ export {
type PluginLauncherActionDeclarationInput,
type PluginLauncherRenderDeclarationInput,
type PluginLauncherDeclarationInput,
type PluginDatabaseDeclarationInput,
type PluginApiRouteDeclarationInput,
type PluginManifestV1Input,
type InstallPlugin,
type UpsertPluginConfig,

View file

@ -1,7 +1,7 @@
export interface ActivityEvent {
id: string;
companyId: string;
actorType: "agent" | "user" | "system";
actorType: "agent" | "user" | "system" | "plugin";
actorId: string;
action: string;
entityType: string;

View file

@ -244,8 +244,13 @@ export type {
PluginLauncherDeclaration,
PluginMinimumHostVersion,
PluginUiDeclaration,
PluginDatabaseDeclaration,
PluginApiRouteCompanyResolution,
PluginApiRouteDeclaration,
PaperclipPluginManifestV1,
PluginRecord,
PluginDatabaseNamespaceRecord,
PluginMigrationRecord,
PluginStateRecord,
PluginConfig,
PluginEntityRecord,
@ -253,4 +258,8 @@ export type {
PluginJobRecord,
PluginJobRunRecord,
PluginWebhookDeliveryRecord,
PluginDatabaseCoreReadTable,
PluginDatabaseMigrationStatus,
PluginDatabaseNamespaceMode,
PluginDatabaseNamespaceStatus,
} from "./plugin.js";

View file

@ -9,6 +9,13 @@ import type {
PluginLauncherAction,
PluginLauncherBounds,
PluginLauncherRenderEnvironment,
PluginApiRouteAuthMode,
PluginApiRouteCheckoutPolicy,
PluginApiRouteMethod,
PluginDatabaseCoreReadTable,
PluginDatabaseMigrationStatus,
PluginDatabaseNamespaceMode,
PluginDatabaseNamespaceStatus,
} from "../constants.js";
// ---------------------------------------------------------------------------
@ -21,6 +28,13 @@ import type {
*/
export type JsonSchema = Record<string, unknown>;
export type {
PluginDatabaseCoreReadTable,
PluginDatabaseMigrationStatus,
PluginDatabaseNamespaceMode,
PluginDatabaseNamespaceStatus,
} from "../constants.js";
// ---------------------------------------------------------------------------
// Manifest sub-types — nested declarations within PaperclipPluginManifestV1
// ---------------------------------------------------------------------------
@ -190,6 +204,44 @@ export interface PluginUiDeclaration {
launchers?: PluginLauncherDeclaration[];
}
/**
* Declares restricted database access for trusted orchestration plugins.
*
* The host derives the final namespace from the plugin key and optional slug,
* applies SQL migrations before worker startup, and gates runtime SQL through
* the `database.namespace.*` capabilities.
*/
export interface PluginDatabaseDeclaration {
/** Optional stable human-readable slug included in the host-derived namespace. */
namespaceSlug?: string;
/** SQL migration directory relative to the plugin package root. */
migrationsDir: string;
/** Public core tables this plugin may read or join at runtime. */
coreReadTables?: PluginDatabaseCoreReadTable[];
}
export type PluginApiRouteCompanyResolution =
| { from: "body"; key: string }
| { from: "query"; key: string }
| { from: "issue"; param: string };
export interface PluginApiRouteDeclaration {
/** Stable plugin-defined route key passed to the worker. */
routeKey: string;
/** HTTP method accepted by this route. */
method: PluginApiRouteMethod;
/** Plugin-local path under `/api/plugins/:pluginId/api`, e.g. `/issues/:issueId/smoke`. */
path: string;
/** Actor class allowed to call the route. */
auth: PluginApiRouteAuthMode;
/** Capability required to expose the route. Currently `api.routes.register`. */
capability: "api.routes.register";
/** Optional checkout policy enforced by the host before worker dispatch. */
checkoutPolicy?: PluginApiRouteCheckoutPolicy;
/** How the host resolves company access for this route. */
companyResolution?: PluginApiRouteCompanyResolution;
}
// ---------------------------------------------------------------------------
// Plugin Manifest V1
// ---------------------------------------------------------------------------
@ -240,6 +292,10 @@ export interface PaperclipPluginManifestV1 {
webhooks?: PluginWebhookDeclaration[];
/** Agent tools this plugin contributes. Requires `agent.tools.register` capability. */
tools?: PluginToolDeclaration[];
/** Restricted plugin-owned database namespace declaration. */
database?: PluginDatabaseDeclaration;
/** Scoped JSON API routes mounted under `/api/plugins/:pluginId/api/*`. */
apiRoutes?: PluginApiRouteDeclaration[];
/**
* Legacy top-level launcher declarations.
* Prefer `ui.launchers` for new manifests.
@ -286,6 +342,31 @@ export interface PluginRecord {
updatedAt: Date;
}
export interface PluginDatabaseNamespaceRecord {
id: string;
pluginId: string;
pluginKey: string;
namespaceName: string;
namespaceMode: PluginDatabaseNamespaceMode;
status: PluginDatabaseNamespaceStatus;
createdAt: Date;
updatedAt: Date;
}
export interface PluginMigrationRecord {
id: string;
pluginId: string;
pluginKey: string;
namespaceName: string;
migrationKey: string;
checksum: string;
pluginVersion: string;
status: PluginDatabaseMigrationStatus;
startedAt: Date;
appliedAt: Date | null;
errorMessage: string | null;
}
// ---------------------------------------------------------------------------
// Plugin State represents a row in the `plugin_state` table
// ---------------------------------------------------------------------------

View file

@ -299,6 +299,8 @@ export {
pluginLauncherActionDeclarationSchema,
pluginLauncherRenderDeclarationSchema,
pluginLauncherDeclarationSchema,
pluginDatabaseDeclarationSchema,
pluginApiRouteDeclarationSchema,
pluginManifestV1Schema,
installPluginSchema,
upsertPluginConfigSchema,
@ -315,6 +317,8 @@ export {
type PluginLauncherActionDeclarationInput,
type PluginLauncherRenderDeclarationInput,
type PluginLauncherDeclarationInput,
type PluginDatabaseDeclarationInput,
type PluginApiRouteDeclarationInput,
type PluginManifestV1Input,
type InstallPlugin,
type UpsertPluginConfig,

View file

@ -11,6 +11,10 @@ import {
PLUGIN_LAUNCHER_BOUNDS,
PLUGIN_LAUNCHER_RENDER_ENVIRONMENTS,
PLUGIN_STATE_SCOPE_KINDS,
PLUGIN_DATABASE_CORE_READ_TABLES,
PLUGIN_API_ROUTE_AUTH_MODES,
PLUGIN_API_ROUTE_CHECKOUT_POLICIES,
PLUGIN_API_ROUTE_METHODS,
} from "../constants.js";
// ---------------------------------------------------------------------------
@ -336,6 +340,48 @@ export const pluginLauncherDeclarationSchema = z.object({
export type PluginLauncherDeclarationInput = z.infer<typeof pluginLauncherDeclarationSchema>;
export const pluginDatabaseDeclarationSchema = z.object({
namespaceSlug: z.string().regex(/^[a-z0-9][a-z0-9_]*$/, {
message: "namespaceSlug must be lowercase letters, digits, or underscores and start with a letter or digit",
}).max(40).optional(),
migrationsDir: z.string().min(1).refine(
(value) => !value.startsWith("/") && !value.includes("..") && !/[\\]/.test(value),
{ message: "migrationsDir must be a relative package path without '..' or backslashes" },
),
coreReadTables: z.array(z.enum(PLUGIN_DATABASE_CORE_READ_TABLES)).optional(),
});
export type PluginDatabaseDeclarationInput = z.infer<typeof pluginDatabaseDeclarationSchema>;
export const pluginApiRouteDeclarationSchema = z.object({
routeKey: z.string().min(1).max(100).regex(/^[a-z0-9][a-z0-9._:-]*$/, {
message: "routeKey must be lowercase letters, digits, dots, colons, underscores, or hyphens",
}),
method: z.enum(PLUGIN_API_ROUTE_METHODS),
path: z.string().min(1).regex(/^\/[a-zA-Z0-9:_./-]*$/, {
message: "path must start with / and contain only path-safe literal or :param segments",
}).refine(
(value) =>
!value.includes("..") &&
!value.includes("//") &&
value !== "/api" &&
!value.startsWith("/api/") &&
value !== "/plugins" &&
!value.startsWith("/plugins/"),
{ message: "path must stay inside the plugin api namespace" },
),
auth: z.enum(PLUGIN_API_ROUTE_AUTH_MODES),
capability: z.literal("api.routes.register"),
checkoutPolicy: z.enum(PLUGIN_API_ROUTE_CHECKOUT_POLICIES).optional(),
companyResolution: z.discriminatedUnion("from", [
z.object({ from: z.literal("body"), key: z.string().min(1) }),
z.object({ from: z.literal("query"), key: z.string().min(1) }),
z.object({ from: z.literal("issue"), param: z.string().min(1) }),
]).optional(),
});
export type PluginApiRouteDeclarationInput = z.infer<typeof pluginApiRouteDeclarationSchema>;
// ---------------------------------------------------------------------------
// Plugin Manifest V1 schema
// ---------------------------------------------------------------------------
@ -405,6 +451,8 @@ export const pluginManifestV1Schema = z.object({
jobs: z.array(pluginJobDeclarationSchema).optional(),
webhooks: z.array(pluginWebhookDeclarationSchema).optional(),
tools: z.array(pluginToolDeclarationSchema).optional(),
database: pluginDatabaseDeclarationSchema.optional(),
apiRoutes: z.array(pluginApiRouteDeclarationSchema).optional(),
launchers: z.array(pluginLauncherDeclarationSchema).optional(),
ui: z.object({
slots: z.array(pluginUiSlotDeclarationSchema).min(1).optional(),
@ -474,6 +522,42 @@ export const pluginManifestV1Schema = z.object({
}
}
if (manifest.apiRoutes && manifest.apiRoutes.length > 0) {
if (!manifest.capabilities.includes("api.routes.register")) {
ctx.addIssue({
code: z.ZodIssueCode.custom,
message: "Capability 'api.routes.register' is required when apiRoutes are declared",
path: ["capabilities"],
});
}
}
if (manifest.database) {
const requiredCapabilities = [
"database.namespace.migrate",
"database.namespace.read",
] as const;
for (const capability of requiredCapabilities) {
if (!manifest.capabilities.includes(capability)) {
ctx.addIssue({
code: z.ZodIssueCode.custom,
message: `Capability '${capability}' is required when database migrations are declared`,
path: ["capabilities"],
});
}
}
const coreReadTables = manifest.database.coreReadTables ?? [];
const duplicates = coreReadTables.filter((table, i) => coreReadTables.indexOf(table) !== i);
if (duplicates.length > 0) {
ctx.addIssue({
code: z.ZodIssueCode.custom,
message: `Duplicate database coreReadTables: ${[...new Set(duplicates)].join(", ")}`,
path: ["database", "coreReadTables"],
});
}
}
// ── Uniqueness checks ──────────────────────────────────────────────────
// Duplicate keys within a plugin's own manifest are always a bug. The host
// would not know which declaration takes precedence, so we reject early.
@ -504,6 +588,27 @@ export const pluginManifestV1Schema = z.object({
}
}
if (manifest.apiRoutes) {
const routeKeys = manifest.apiRoutes.map((route) => route.routeKey);
const duplicateKeys = routeKeys.filter((key, i) => routeKeys.indexOf(key) !== i);
if (duplicateKeys.length > 0) {
ctx.addIssue({
code: z.ZodIssueCode.custom,
message: `Duplicate api route keys: ${[...new Set(duplicateKeys)].join(", ")}`,
path: ["apiRoutes"],
});
}
const routeSignatures = manifest.apiRoutes.map((route) => `${route.method} ${route.path}`);
const duplicateRoutes = routeSignatures.filter((sig, i) => routeSignatures.indexOf(sig) !== i);
if (duplicateRoutes.length > 0) {
ctx.addIssue({
code: z.ZodIssueCode.custom,
message: `Duplicate api routes: ${[...new Set(duplicateRoutes)].join(", ")}`,
path: ["apiRoutes"],
});
}
}
// tool names must be unique within the plugin (namespaced at runtime)
if (manifest.tools) {
const toolNames = manifest.tools.map((t) => t.name);