Add secrets provider vaults and remote import (#5429)

## Thinking Path

> - Paperclip orchestrates AI-agent companies and needs secrets handling
to work across local development, hosted operators, and governed agent
execution.
> - The affected subsystem is the company-scoped secrets control plane:
database schema, server services/routes, CLI workflows, and the Secrets
settings UI.
> - The gap was that secrets were local-only and operators could not
manage provider vaults or import existing remote references without
exposing plaintext.
> - This branch adds provider vault configuration plus an AWS Secrets
Manager remote-import path while preserving company boundaries, binding
context, and audit trails.
> - I kept the PR to a single branch PR, removed unrelated
lockfile/package drift, rebased the full branch onto the current
`public-gh/master`, and addressed fresh Greptile findings.
> - The benefit is a reviewable implementation of provider-backed
secrets with focused tests covering provider selection, import
conflicts, deleted secret reuse, rotation guards, and AWS signing
behavior.

## What Changed

- Added provider vault support for company secrets, including provider
config storage, default vault handling, health checks, binding usage,
access events, and remote import preview/commit.
- Added an AWS Secrets Manager provider using SigV4 request signing,
bounded request timeouts, namespace guardrails, cached runtime
credential resolution, and external-reference linking without plaintext
reads.
- Added Secrets UI surfaces for vault management and remote import, plus
CLI/API documentation for setup and operations.
- Stabilized routine webhook secret binding paths and SSH
environment-driver fixture bindings discovered during verification.
- Addressed Greptile and CI findings: no lockfile/package drift,
monotonic migration metadata, disabled-vault default races, soft-deleted
secret hiding/recreate behavior, remove behavior with disabled vaults,
soft-deleted external-reference re-import, non-active rotation guards,
managed-secret soft deletion through PATCH, and per-call AWS SDK
credential client churn.
- Rebased this branch onto `public-gh/master` at `0e1a5828` and
force-pushed with lease to keep this as the single PR for the branch.

## Verification

- `git fetch public-gh master`
- `git rebase public-gh/master`
- `git diff --name-only public-gh/master...HEAD | grep
'^pnpm-lock\.yaml$' || true` confirmed `pnpm-lock.yaml` is not in the PR
diff.
- Confirmed migration ordering: master ends at `0081_optimal_dormammu`;
this PR adds `0082_dry_vision` and
`0083_company_secret_provider_configs`.
- Inspected migrations for repeat safety: new tables/indexes use `IF NOT
EXISTS`; foreign keys are guarded by `DO $$ ... IF NOT EXISTS`; column
additions use `ADD COLUMN IF NOT EXISTS`.
- `pnpm -r typecheck` passed before the Greptile follow-up commits.
- `pnpm test:run` ran the full stable Vitest path before the Greptile
follow-up commits; it completed with 3 timing-related failures under
parallel load: `codex-local-execute.test.ts`,
`cursor-local-execute.test.ts`, and `environment-service.test.ts`.
- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/codex-local-execute.test.ts
src/__tests__/cursor-local-execute.test.ts
src/__tests__/environment-service.test.ts` passed on targeted rerun
(`24/24`).
- `pnpm build` passed before the Greptile follow-up commits. Vite
reported existing chunk-size/dynamic-import warnings.
- After Greptile follow-up commits: `pnpm --filter @paperclipai/server
exec vitest run src/__tests__/secrets-service.test.ts` passed (`26/26`).
- After Greptile follow-up commits: `pnpm --filter @paperclipai/server
exec vitest run src/__tests__/aws-secrets-manager-provider.test.ts
src/__tests__/secrets-service.test.ts` passed (`39/39`).
- After Greptile follow-up commits: `pnpm --filter @paperclipai/server
typecheck` passed.
- Captured Storybook screenshots from `ui/storybook-static` for visual
review.
- Latest PR checks on `5ca3a5cf`: `policy`, serialized server suites
1/4-4/4, `Canary Dry Run`, `e2e`, `security/snyk`, and `Greptile Review`
pass; aggregate `verify` is still registering the completed child
checks.
- Greptile review loop continued through the latest requested pass; all
Greptile review threads are resolved and the latest `Greptile Review`
check on `5ca3a5cf` passed with 0 comments added.

## Screenshots

Before: the provider-vault and remote-import surfaces did not exist on
`master`; these are after-state screenshots from the Storybook fixtures.

![Secrets
inventory](https://raw.githubusercontent.com/paperclipai/paperclip/PAP-2339-secrets-make-a-plan/doc/pr/5429/secrets-inventory.png)

![Secret binding
picker](https://raw.githubusercontent.com/paperclipai/paperclip/PAP-2339-secrets-make-a-plan/doc/pr/5429/secret-binding-picker.png)

![Environment editor with
secrets](https://raw.githubusercontent.com/paperclipai/paperclip/PAP-2339-secrets-make-a-plan/doc/pr/5429/env-editor-with-secrets.png)

## Risks

- Migration risk: this adds new secret provider tables and extends
existing secret rows. The migrations were checked for monotonic ordering
and idempotent guards, but reviewers should still inspect upgrade
behavior carefully.
- Provider risk: AWS support uses direct SigV4 requests. Automated tests
cover signing, request timeouts, vault-config selection, namespace
guardrails, pending-version archival, sanitized provider errors, and
service-level cleanup paths. A real-vault AWS smoke test remains
deployment validation for an operator with AWS credentials rather than
an unverified merge blocker in this local branch.
- UI risk: the Secrets page and import dialog are large new surfaces;
screenshots are included above for reviewer inspection.
- Verification risk: the full local stable test command hit
parallel-load timing failures, although the exact failed files passed
when rerun directly.
- Operational risk: remote import intentionally avoids plaintext reads;
operators must understand that imported external references resolve at
runtime and may fail if AWS permissions change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- OpenAI Codex, GPT-5 coding agent with local shell/tool use in the
Paperclip worktree. Exact context-window size was not exposed by the
runtime.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] If this change affects the UI, I have included before/after
screenshots
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

packages/shared/src/validators/index.ts

@@ -282,9 +282,27 @@ export {
   envBindingSchema,
   envConfigSchema,
   createSecretSchema,
+  createSecretProviderConfigSchema,
+  updateSecretProviderConfigSchema,
+  remoteSecretImportPreviewSchema,
+  remoteSecretImportSchema,
+  remoteSecretImportSelectionSchema,
+  localEncryptedProviderConfigSchema,
+  awsSecretsManagerProviderConfigSchema,
+  gcpSecretManagerProviderConfigSchema,
+  vaultProviderConfigSchema,
+  secretProviderConfigPayloadSchema,
+  createSecretBindingSchema,
   rotateSecretSchema,
+  secretBindingTargetSchema,
   updateSecretSchema,
+  type CreateSecretBinding,
   type CreateSecret,
+  type CreateSecretProviderConfig,
+  type UpdateSecretProviderConfig,
+  type RemoteSecretImportPreview,
+  type RemoteSecretImport,
+  type RemoteSecretImportSelection,
   type RotateSecret,
   type UpdateSecret,
 } from "./secret.js";

packages/shared/src/validators/secret.test.ts

@@ -0,0 +1,157 @@
+import { describe, expect, it } from "vitest";
+import {
+  createSecretProviderConfigSchema,
+  createSecretSchema,
+  remoteSecretImportPreviewSchema,
+  remoteSecretImportSchema,
+  secretProviderConfigPayloadSchema,
+  updateSecretProviderConfigSchema,
+} from "./secret.js";
+
+describe("secret validators", () => {
+  it("rejects externalRef on managed secrets", () => {
+    expect(() =>
+      createSecretSchema.parse({
+        name: "OpenAI API Key",
+        managedMode: "paperclip_managed",
+        value: "secret-value",
+        externalRef: "arn:aws:secretsmanager:us-east-1:123456789012:secret:shared/other",
+      }),
+    ).toThrow(/Managed secrets cannot set externalRef/);
+  });
+
+  it("allows externalRef on external reference secrets", () => {
+    const parsed = createSecretSchema.parse({
+      name: "Shared Secret",
+      managedMode: "external_reference",
+      externalRef: "arn:aws:secretsmanager:us-east-1:123456789012:secret:shared/other",
+    });
+
+    expect(parsed.externalRef).toContain(":secret:shared/other");
+  });
+
+  it("accepts non-sensitive local and AWS provider vault metadata", () => {
+    expect(() =>
+      createSecretProviderConfigSchema.parse({
+        provider: "local_encrypted",
+        displayName: "Local",
+        config: { backupReminderAcknowledged: true },
+      }),
+    ).not.toThrow();
+
+    expect(() =>
+      createSecretProviderConfigSchema.parse({
+        provider: "aws_secrets_manager",
+        displayName: "AWS",
+        config: {
+          region: "us-east-1",
+          namespace: "production",
+          secretNamePrefix: "paperclip",
+        },
+      }),
+    ).not.toThrow();
+  });
+
+  it("accepts origin-only Vault provider vault addresses", () => {
+    expect(() =>
+      createSecretProviderConfigSchema.parse({
+        provider: "vault",
+        displayName: "Vault draft",
+        config: { address: " https://vault.example.com/ " },
+      }),
+    ).not.toThrow();
+
+    const parsed = secretProviderConfigPayloadSchema.parse({
+      provider: "vault",
+      config: { address: " https://vault.example.com/ " },
+    });
+
+    expect(parsed.provider).toBe("vault");
+    if (parsed.provider !== "vault") throw new Error("Expected vault provider payload");
+    expect(parsed.config.address).toBe("https://vault.example.com");
+  });
+
+  it.each([
+    "https://user:pass@vault.example.com",
+    "https://vault.example.com?token=hvs.x",
+    "https://vault.example.com#token=hvs.x",
+    "https://vault.example.com/v1/secret",
+  ])("rejects credential-bearing or non-origin Vault addresses: %s", (address) => {
+    expect(() =>
+      createSecretProviderConfigSchema.parse({
+        provider: "vault",
+        displayName: "Vault draft",
+        config: { address },
+      }),
+    ).toThrow(/origin-only HTTP\(S\) URL/i);
+  });
+
+  it("rejects unsafe Vault addresses in provider payload validation used by updates", () => {
+    expect(() =>
+      secretProviderConfigPayloadSchema.parse({
+        provider: "vault",
+        config: { address: "https://vault.example.com?client_token=hvs.x" },
+      }),
+    ).toThrow(/origin-only HTTP\(S\) URL/i);
+  });
+
+  it("rejects unsafe Vault addresses in provider vault update payloads", () => {
+    expect(() =>
+      updateSecretProviderConfigSchema.parse({
+        config: { address: "https://vault.example.com#token=hvs.x" },
+      }),
+    ).toThrow(/origin-only HTTP\(S\) URL/i);
+  });
+
+  it("validates AWS remote import preview and import payloads", () => {
+    expect(
+      remoteSecretImportPreviewSchema.parse({
+        providerConfigId: "11111111-1111-4111-8111-111111111111",
+        query: "openai",
+        pageSize: 50,
+      }),
+    ).toEqual({
+      providerConfigId: "11111111-1111-4111-8111-111111111111",
+      query: "openai",
+      pageSize: 50,
+    });
+
+    expect(
+      remoteSecretImportSchema.parse({
+        providerConfigId: "11111111-1111-4111-8111-111111111111",
+        secrets: [
+          {
+            externalRef: "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/openai",
+            name: "OpenAI API key",
+            key: "OPENAI_API_KEY",
+            description: "  Operator-entered Paperclip description  ",
+            providerMetadata: { name: "prod/openai" },
+          },
+        ],
+      }),
+    ).toMatchObject({
+      providerConfigId: "11111111-1111-4111-8111-111111111111",
+      secrets: [
+        expect.objectContaining({
+          key: "OPENAI_API_KEY",
+          description: "Operator-entered Paperclip description",
+        }),
+      ],
+    });
+  });
+
+  it("caps AWS remote import paging and row counts", () => {
+    expect(() =>
+      remoteSecretImportPreviewSchema.parse({
+        providerConfigId: "11111111-1111-4111-8111-111111111111",
+        pageSize: 101,
+      }),
+    ).toThrow();
+    expect(() =>
+      remoteSecretImportSchema.parse({
+        providerConfigId: "11111111-1111-4111-8111-111111111111",
+        secrets: [],
+      }),
+    ).toThrow();
+  });
+});

packages/shared/src/validators/secret.ts

@@ -1,5 +1,11 @@
 import { z } from "zod";
-import { SECRET_PROVIDERS } from "../constants.js";
+import {
+  SECRET_BINDING_TARGET_TYPES,
+  SECRET_MANAGED_MODES,
+  SECRET_PROVIDER_CONFIG_STATUSES,
+  SECRET_PROVIDERS,
+  SECRET_STATUSES,
+} from "../constants.js";
 
 export const envBindingPlainSchema = z.object({
   type: z.literal("plain"),
@@ -23,25 +29,252 @@ export const envConfigSchema = z.record(envBindingSchema);
 
 export const createSecretSchema = z.object({
   name: z.string().min(1),
+  key: z.string().min(1).regex(/^[a-zA-Z0-9_.-]+$/).optional(),
   provider: z.enum(SECRET_PROVIDERS).optional(),
-  value: z.string().min(1),
+  providerConfigId: z.string().uuid().optional().nullable(),
+  managedMode: z.enum(SECRET_MANAGED_MODES).optional(),
+  value: z.string().min(1).optional().nullable(),
   description: z.string().optional().nullable(),
   externalRef: z.string().optional().nullable(),
+  providerMetadata: z.record(z.unknown()).optional().nullable(),
+  providerVersionRef: z.string().optional().nullable(),
+}).superRefine((value, ctx) => {
+  if ((value.managedMode ?? "paperclip_managed") === "external_reference") {
+    if (!value.externalRef?.trim()) {
+      ctx.addIssue({
+        code: z.ZodIssueCode.custom,
+        path: ["externalRef"],
+        message: "External reference secrets require externalRef",
+      });
+    }
+    return;
+  }
+  if (value.externalRef?.trim()) {
+    ctx.addIssue({
+      code: z.ZodIssueCode.custom,
+      path: ["externalRef"],
+      message: "Managed secrets cannot set externalRef",
+    });
+  }
+  if (!value.value?.trim()) {
+    ctx.addIssue({
+      code: z.ZodIssueCode.custom,
+      path: ["value"],
+      message: "Managed secrets require value",
+    });
+  }
 });
 
 export type CreateSecret = z.infer<typeof createSecretSchema>;
 
 export const rotateSecretSchema = z.object({
-  value: z.string().min(1),
+  value: z.string().min(1).optional().nullable(),
   externalRef: z.string().optional().nullable(),
+  providerVersionRef: z.string().optional().nullable(),
+  providerConfigId: z.string().uuid().optional().nullable(),
 });
 
 export type RotateSecret = z.infer<typeof rotateSecretSchema>;
 
 export const updateSecretSchema = z.object({
   name: z.string().min(1).optional(),
+  key: z.string().min(1).regex(/^[a-zA-Z0-9_.-]+$/).optional(),
+  status: z.enum(SECRET_STATUSES).optional(),
+  providerConfigId: z.string().uuid().optional().nullable(),
   description: z.string().optional().nullable(),
   externalRef: z.string().optional().nullable(),
+  providerMetadata: z.record(z.unknown()).optional().nullable(),
 });
 
 export type UpdateSecret = z.infer<typeof updateSecretSchema>;
+
+export const secretBindingTargetSchema = z.object({
+  targetType: z.enum(SECRET_BINDING_TARGET_TYPES),
+  targetId: z.string().min(1),
+  configPath: z.string().min(1),
+});
+
+export const createSecretBindingSchema = secretBindingTargetSchema.extend({
+  secretId: z.string().uuid(),
+  versionSelector: z.union([z.literal("latest"), z.number().int().positive()]).default("latest"),
+  required: z.boolean().default(true),
+  label: z.string().optional().nullable(),
+});
+
+export type CreateSecretBinding = z.infer<typeof createSecretBindingSchema>;
+
+const safeShortText = z.string().trim().min(1).max(160);
+const optionalSafeShortText = safeShortText.optional().nullable();
+
+const deniedProviderConfigKeyPattern =
+  /^(access[-_]?key([-_]?id)?|secret[-_]?access[-_]?key|secret[-_]?key|token|password|passwd|credential|credentials|private[-_]?key|pem|jwt|session[-_]?token|service[-_]?account([-_]?json)?|client[-_]?secret|secret[-_]?id|unseal[-_]?key|recovery[-_]?key|key[-_]?file([-_]?path)?|token[-_]?file([-_]?path)?)$/i;
+
+function rejectSensitiveProviderConfigKeys(value: unknown, ctx: z.RefinementCtx) {
+  if (!value || typeof value !== "object" || Array.isArray(value)) return;
+  for (const key of Object.keys(value)) {
+    if (!deniedProviderConfigKeyPattern.test(key)) continue;
+    ctx.addIssue({
+      code: z.ZodIssueCode.custom,
+      path: ["config", key],
+      message: `Provider vault config cannot persist sensitive field: ${key}`,
+    });
+  }
+}
+
+export const localEncryptedProviderConfigSchema = z.object({
+  backupReminderAcknowledged: z.boolean().optional(),
+}).strict();
+
+export const awsSecretsManagerProviderConfigSchema = z.object({
+  region: z.string().trim().regex(/^[a-z]{2}(?:-gov)?-[a-z]+-\d+$/, "Invalid AWS region"),
+  namespace: optionalSafeShortText,
+  secretNamePrefix: optionalSafeShortText,
+  kmsKeyId: z.string().trim().min(1).max(512).optional().nullable(),
+  ownerTag: optionalSafeShortText,
+  environmentTag: optionalSafeShortText,
+}).strict();
+
+export const gcpSecretManagerProviderConfigSchema = z.object({
+  projectId: z.string().trim().min(1).max(128).regex(/^[a-z][a-z0-9-]{4,127}$/).optional().nullable(),
+  location: optionalSafeShortText,
+  namespace: optionalSafeShortText,
+  secretNamePrefix: optionalSafeShortText,
+}).strict();
+
+const vaultAddressSchema = z.preprocess(
+  (value) => typeof value === "string" ? value.trim() : value,
+  z.string().url().superRefine((value, ctx) => {
+    let url: URL;
+    try {
+      url = new URL(value);
+    } catch {
+      return;
+    }
+    const hasPath = url.pathname !== "" && url.pathname !== "/";
+    if (
+      (url.protocol !== "http:" && url.protocol !== "https:") ||
+      url.username ||
+      url.password ||
+      url.search ||
+      url.hash ||
+      hasPath
+    ) {
+      ctx.addIssue({
+        code: z.ZodIssueCode.custom,
+        message: "Vault address must be an origin-only HTTP(S) URL without credentials, path, query, or fragment",
+      });
+    }
+  }).transform((value) => new URL(value).origin),
+);
+
+function rejectUnsafeVaultAddress(value: unknown, ctx: z.RefinementCtx) {
+  if (value === undefined || value === null) return;
+  const parsed = vaultAddressSchema.safeParse(value);
+  if (parsed.success) return;
+  for (const issue of parsed.error.issues) {
+    ctx.addIssue({
+      ...issue,
+      path: ["config", "address", ...issue.path],
+    });
+  }
+}
+
+export const vaultProviderConfigSchema = z.object({
+  address: vaultAddressSchema.optional().nullable(),
+  namespace: optionalSafeShortText,
+  mountPath: optionalSafeShortText,
+  secretPathPrefix: optionalSafeShortText,
+}).strict();
+
+export const secretProviderConfigPayloadSchema = z.discriminatedUnion("provider", [
+  z.object({ provider: z.literal("local_encrypted"), config: localEncryptedProviderConfigSchema }),
+  z.object({ provider: z.literal("aws_secrets_manager"), config: awsSecretsManagerProviderConfigSchema }),
+  z.object({ provider: z.literal("gcp_secret_manager"), config: gcpSecretManagerProviderConfigSchema }),
+  z.object({ provider: z.literal("vault"), config: vaultProviderConfigSchema }),
+]);
+
+export const createSecretProviderConfigSchema = z.object({
+  provider: z.enum(SECRET_PROVIDERS),
+  displayName: z.string().trim().min(1).max(120),
+  status: z.enum(SECRET_PROVIDER_CONFIG_STATUSES).optional(),
+  isDefault: z.boolean().optional(),
+  config: z.record(z.unknown()).default({}),
+}).superRefine((value, ctx) => {
+  rejectSensitiveProviderConfigKeys(value.config, ctx);
+  const parsed = secretProviderConfigPayloadSchema.safeParse({
+    provider: value.provider,
+    config: value.config,
+  });
+  if (!parsed.success) {
+    for (const issue of parsed.error.issues) {
+      ctx.addIssue({
+        ...issue,
+        path: issue.path[0] === "config" ? issue.path : ["config", ...issue.path],
+      });
+    }
+  }
+  const status = value.status ?? (["gcp_secret_manager", "vault"].includes(value.provider) ? "coming_soon" : "ready");
+  if ((value.provider === "gcp_secret_manager" || value.provider === "vault") && status !== "coming_soon" && status !== "disabled") {
+    ctx.addIssue({
+      code: z.ZodIssueCode.custom,
+      path: ["status"],
+      message: `${value.provider} provider vaults are locked while coming soon`,
+    });
+  }
+  if ((status === "coming_soon" || status === "disabled") && value.isDefault) {
+    ctx.addIssue({
+      code: z.ZodIssueCode.custom,
+      path: ["isDefault"],
+      message: "Only ready or warning provider vaults can be default",
+    });
+  }
+});
+
+export type CreateSecretProviderConfig = z.infer<typeof createSecretProviderConfigSchema>;
+
+export const updateSecretProviderConfigSchema = z.object({
+  displayName: z.string().trim().min(1).max(120).optional(),
+  status: z.enum(SECRET_PROVIDER_CONFIG_STATUSES).optional(),
+  isDefault: z.boolean().optional(),
+  config: z.record(z.unknown()).optional(),
+}).superRefine((value, ctx) => {
+  if (value.config !== undefined) {
+    rejectSensitiveProviderConfigKeys(value.config, ctx);
+    rejectUnsafeVaultAddress(value.config.address, ctx);
+  }
+  if ((value.status === "coming_soon" || value.status === "disabled") && value.isDefault) {
+    ctx.addIssue({
+      code: z.ZodIssueCode.custom,
+      path: ["isDefault"],
+      message: "Only ready or warning provider vaults can be default",
+    });
+  }
+});
+
+export type UpdateSecretProviderConfig = z.infer<typeof updateSecretProviderConfigSchema>;
+
+export const remoteSecretImportPreviewSchema = z.object({
+  providerConfigId: z.string().uuid(),
+  query: z.string().trim().max(200).optional().nullable(),
+  nextToken: z.string().trim().min(1).max(4096).optional().nullable(),
+  pageSize: z.number().int().min(1).max(100).optional(),
+});
+
+export type RemoteSecretImportPreview = z.infer<typeof remoteSecretImportPreviewSchema>;
+
+export const remoteSecretImportSelectionSchema = z.object({
+  externalRef: z.string().trim().min(1).max(2048),
+  name: z.string().trim().min(1).max(160).optional().nullable(),
+  key: z.string().trim().min(1).max(120).regex(/^[a-zA-Z0-9_.-]+$/).optional().nullable(),
+  description: z.string().trim().max(500).optional().nullable(),
+  providerVersionRef: z.string().trim().min(1).max(512).optional().nullable(),
+  providerMetadata: z.record(z.unknown()).optional().nullable(),
+});
+
+export const remoteSecretImportSchema = z.object({
+  providerConfigId: z.string().uuid(),
+  secrets: z.array(remoteSecretImportSelectionSchema).min(1).max(100),
+});
+
+export type RemoteSecretImportSelection = z.infer<typeof remoteSecretImportSelectionSchema>;
+export type RemoteSecretImport = z.infer<typeof remoteSecretImportSchema>;