Add secrets provider vaults and remote import (#5429)

## Thinking Path

> - Paperclip orchestrates AI-agent companies and needs secrets handling
to work across local development, hosted operators, and governed agent
execution.
> - The affected subsystem is the company-scoped secrets control plane:
database schema, server services/routes, CLI workflows, and the Secrets
settings UI.
> - The gap was that secrets were local-only and operators could not
manage provider vaults or import existing remote references without
exposing plaintext.
> - This branch adds provider vault configuration plus an AWS Secrets
Manager remote-import path while preserving company boundaries, binding
context, and audit trails.
> - I kept the PR to a single branch PR, removed unrelated
lockfile/package drift, rebased the full branch onto the current
`public-gh/master`, and addressed fresh Greptile findings.
> - The benefit is a reviewable implementation of provider-backed
secrets with focused tests covering provider selection, import
conflicts, deleted secret reuse, rotation guards, and AWS signing
behavior.

## What Changed

- Added provider vault support for company secrets, including provider
config storage, default vault handling, health checks, binding usage,
access events, and remote import preview/commit.
- Added an AWS Secrets Manager provider using SigV4 request signing,
bounded request timeouts, namespace guardrails, cached runtime
credential resolution, and external-reference linking without plaintext
reads.
- Added Secrets UI surfaces for vault management and remote import, plus
CLI/API documentation for setup and operations.
- Stabilized routine webhook secret binding paths and SSH
environment-driver fixture bindings discovered during verification.
- Addressed Greptile and CI findings: no lockfile/package drift,
monotonic migration metadata, disabled-vault default races, soft-deleted
secret hiding/recreate behavior, remove behavior with disabled vaults,
soft-deleted external-reference re-import, non-active rotation guards,
managed-secret soft deletion through PATCH, and per-call AWS SDK
credential client churn.
- Rebased this branch onto `public-gh/master` at `0e1a5828` and
force-pushed with lease to keep this as the single PR for the branch.

## Verification

- `git fetch public-gh master`
- `git rebase public-gh/master`
- `git diff --name-only public-gh/master...HEAD | grep
'^pnpm-lock\.yaml$' || true` confirmed `pnpm-lock.yaml` is not in the PR
diff.
- Confirmed migration ordering: master ends at `0081_optimal_dormammu`;
this PR adds `0082_dry_vision` and
`0083_company_secret_provider_configs`.
- Inspected migrations for repeat safety: new tables/indexes use `IF NOT
EXISTS`; foreign keys are guarded by `DO $$ ... IF NOT EXISTS`; column
additions use `ADD COLUMN IF NOT EXISTS`.
- `pnpm -r typecheck` passed before the Greptile follow-up commits.
- `pnpm test:run` ran the full stable Vitest path before the Greptile
follow-up commits; it completed with 3 timing-related failures under
parallel load: `codex-local-execute.test.ts`,
`cursor-local-execute.test.ts`, and `environment-service.test.ts`.
- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/codex-local-execute.test.ts
src/__tests__/cursor-local-execute.test.ts
src/__tests__/environment-service.test.ts` passed on targeted rerun
(`24/24`).
- `pnpm build` passed before the Greptile follow-up commits. Vite
reported existing chunk-size/dynamic-import warnings.
- After Greptile follow-up commits: `pnpm --filter @paperclipai/server
exec vitest run src/__tests__/secrets-service.test.ts` passed (`26/26`).
- After Greptile follow-up commits: `pnpm --filter @paperclipai/server
exec vitest run src/__tests__/aws-secrets-manager-provider.test.ts
src/__tests__/secrets-service.test.ts` passed (`39/39`).
- After Greptile follow-up commits: `pnpm --filter @paperclipai/server
typecheck` passed.
- Captured Storybook screenshots from `ui/storybook-static` for visual
review.
- Latest PR checks on `5ca3a5cf`: `policy`, serialized server suites
1/4-4/4, `Canary Dry Run`, `e2e`, `security/snyk`, and `Greptile Review`
pass; aggregate `verify` is still registering the completed child
checks.
- Greptile review loop continued through the latest requested pass; all
Greptile review threads are resolved and the latest `Greptile Review`
check on `5ca3a5cf` passed with 0 comments added.

## Screenshots

Before: the provider-vault and remote-import surfaces did not exist on
`master`; these are after-state screenshots from the Storybook fixtures.

![Secrets
inventory](https://raw.githubusercontent.com/paperclipai/paperclip/PAP-2339-secrets-make-a-plan/doc/pr/5429/secrets-inventory.png)

![Secret binding
picker](https://raw.githubusercontent.com/paperclipai/paperclip/PAP-2339-secrets-make-a-plan/doc/pr/5429/secret-binding-picker.png)

![Environment editor with
secrets](https://raw.githubusercontent.com/paperclipai/paperclip/PAP-2339-secrets-make-a-plan/doc/pr/5429/env-editor-with-secrets.png)

## Risks

- Migration risk: this adds new secret provider tables and extends
existing secret rows. The migrations were checked for monotonic ordering
and idempotent guards, but reviewers should still inspect upgrade
behavior carefully.
- Provider risk: AWS support uses direct SigV4 requests. Automated tests
cover signing, request timeouts, vault-config selection, namespace
guardrails, pending-version archival, sanitized provider errors, and
service-level cleanup paths. A real-vault AWS smoke test remains
deployment validation for an operator with AWS credentials rather than
an unverified merge blocker in this local branch.
- UI risk: the Secrets page and import dialog are large new surfaces;
screenshots are included above for reviewer inspection.
- Verification risk: the full local stable test command hit
parallel-load timing failures, although the exact failed files passed
when rerun directly.
- Operational risk: remote import intentionally avoids plaintext reads;
operators must understand that imported external references resolve at
runtime and may fail if AWS permissions change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- OpenAI Codex, GPT-5 coding agent with local shell/tool use in the
Paperclip worktree. Exact context-window size was not exposed by the
runtime.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] If this change affects the UI, I have included before/after
screenshots
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Dotta 2026-05-09 18:22:17 -05:00 committed by GitHub
parent 06e6ee25cd
commit 778e775c35
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
103 changed files with 16971 additions and 509 deletions

View file

@ -0,0 +1,133 @@
---
title: Secrets Remote Import
summary: AWS Secrets Manager metadata-only remote import API
---
Remote import lets the board link existing AWS Secrets Manager entries as
Paperclip `external_reference` secrets without copying plaintext into
Paperclip.
Both routes are board-only and company-scoped. The selected provider vault must
belong to the company, use `aws_secrets_manager`, and have a selectable status
(`ready` or `warning`). Disabled, coming-soon, or cross-company vaults are
rejected.
Remote import is an inventory and metadata workflow. Preview calls AWS
`ListSecrets` only and import stores a Paperclip external reference plus
fingerprint/version metadata. Neither route calls `GetSecretValue` or
`BatchGetSecretValue`, requests `SecretString`, requires KMS decrypt, logs raw
remote metadata, or copies secret plaintext into Paperclip.
## Preview Remote AWS Secrets
```
POST /api/companies/{companyId}/secrets/remote-import/preview
{
"providerConfigId": "<aws-vault-uuid>",
"query": "stripe",
"nextToken": "optional-provider-page-token",
"pageSize": 50
}
```
`query` is optional and is sent to AWS as an inventory filter. Treat it as
non-secret metadata because AWS may record list request parameters in
CloudTrail. `nextToken` is an opaque AWS cursor; pass it back unchanged.
`pageSize` is capped at 100.
Response:
```json
{
"providerConfigId": "<aws-vault-uuid>",
"provider": "aws_secrets_manager",
"nextToken": null,
"candidates": [
{
"externalRef": "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/stripe",
"remoteName": "prod/stripe",
"name": "prod/stripe",
"key": "prod-stripe",
"providerVersionRef": null,
"providerMetadata": {
"lastChangedDate": "2026-05-06T00:00:00.000Z",
"hasDescription": true
},
"status": "ready",
"importable": true,
"conflicts": []
}
]
}
```
Candidate `status` values:
- `ready`: no existing exact external reference and no name/key collision.
- `duplicate`: an existing secret already has the exact provider `externalRef`.
- `conflict`: the suggested Paperclip `name` or `key` is already in use.
Conflict `type` values are `exact_reference`, `name`, `key`, and
`provider_guardrail`. AWS refs under Paperclip's own managed namespace are
blocked as external references so one company cannot import another company's
Paperclip-managed AWS secret through a broad runtime role.
## Import Remote AWS Secret References
```
POST /api/companies/{companyId}/secrets/remote-import
{
"providerConfigId": "<aws-vault-uuid>",
"secrets": [
{
"externalRef": "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/stripe",
"name": "Stripe production key",
"key": "stripe-production-key",
"description": "Stripe key used by production checkout",
"providerVersionRef": null,
"providerMetadata": {
"lastChangedDate": "2026-05-06T00:00:00.000Z",
"hasDescription": true
}
}
]
}
```
The import response is row-level. Ready rows become active
`external_reference` secrets with version metadata only. Exact-reference
duplicates and name/key conflicts are skipped without failing the whole request.
The `secrets` array accepts 1-100 rows, and the backend re-checks duplicates and
conflicts at submit time.
Each row may include an optional Paperclip `description` entered during review;
blank descriptions are stored as `null`. AWS provider descriptions are not
copied into this field.
```json
{
"providerConfigId": "<aws-vault-uuid>",
"provider": "aws_secrets_manager",
"importedCount": 1,
"skippedCount": 1,
"errorCount": 0,
"results": [
{
"externalRef": "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/stripe",
"name": "Stripe production key",
"key": "stripe-production-key",
"status": "imported",
"reason": null,
"secretId": "<paperclip-secret-id>",
"conflicts": []
}
]
}
```
Activity logs record aggregate counts and provider/vault ids only, not remote
secret names, ARNs, tags, or values.
Imported references may still fail during a future bound runtime resolution if
the Paperclip runtime role can list the AWS secret but lacks
`secretsmanager:GetSecretValue` or required KMS decrypt permission for that
specific secret.

View file

@ -25,16 +25,357 @@ POST /api/companies/{companyId}/secrets
The value is encrypted at rest. Only the secret ID and metadata are returned.
## Update Secret
To link a provider-owned secret without copying the value into Paperclip, create
an external-reference secret:
```json
{
"name": "prod-stripe-key",
"provider": "aws_secrets_manager",
"managedMode": "external_reference",
"externalRef": "arn:aws:secretsmanager:us-east-1:123456789012:secret:paperclip/prod/stripe",
"providerVersionRef": "version-id-or-label"
}
```
Paperclip stores the provider reference and a non-sensitive fingerprint only.
The value is resolved, when the provider is configured, through the server
runtime path that enforces binding context and records access events.
## Provider Health
```
PATCH /api/secrets/{secretId}
GET /api/companies/{companyId}/secret-providers/health
```
Returns provider setup diagnostics, warnings, and local backup guidance. Health
responses must not include secret values or provider credentials.
For `aws_secrets_manager`, an unready health response names the missing
non-secret provider environment variables, the AWS SDK default credential source
expected by the server runtime, and the custody rule that AWS bootstrap
credentials must not be stored in Paperclip `company_secrets`.
The equivalent CLI check is:
```sh
pnpm paperclipai secrets doctor --company-id {companyId}
```
## Provider Vaults
Provider vaults are named, company-scoped configurations that route secret
material to one of the supported provider backends. See the
[secrets deploy guide](/deploy/secrets#provider-vaults) for the operator model
and custody rules.
All routes below require board auth and company access. Mutating routes emit
`secret_provider_config.*` activity-log entries. No route in this surface
returns provider credential values; submitting credential-shaped fields in
`config` is rejected at validation time.
### List Vaults
```
GET /api/companies/{companyId}/secret-provider-configs
```
Returns every vault for the company (including disabled rows for audit), each
with id, provider, displayName, status, isDefault, non-sensitive `config`,
latest health snapshot (`healthStatus`, `healthCheckedAt`, `healthMessage`,
`healthDetails`), `disabledAt`, and audit columns.
### Create Vault
```
POST /api/companies/{companyId}/secret-provider-configs
{
"provider": "aws_secrets_manager",
"displayName": "Prod US-East",
"isDefault": true,
"config": {
"region": "us-east-1",
"namespace": "paperclip",
"secretNamePrefix": "paperclip",
"kmsKeyId": "arn:aws:kms:us-east-1:123456789012:key/abcd-...",
"environmentTag": "production"
}
}
```
Per-provider `config` shapes:
- `local_encrypted`: optional `backupReminderAcknowledged: boolean`.
- `aws_secrets_manager`: required `region`; optional `namespace`,
`secretNamePrefix`, `kmsKeyId`, `ownerTag`, `environmentTag`.
- `gcp_secret_manager` (coming soon): optional `projectId`, `location`,
`namespace`, `secretNamePrefix`.
- `vault` (coming soon): optional origin-only HTTPS `address`, `namespace`,
`mountPath`, `secretPathPrefix`. `address` values with embedded credentials,
paths, query strings, or fragments are rejected.
`status` defaults to `ready` for `local_encrypted` and `aws_secrets_manager`,
and to `coming_soon` for `gcp_secret_manager` and `vault`. Coming-soon and
disabled vaults cannot be marked `isDefault`. Setting `isDefault: true` clears
the previous default for the same provider in the same transaction.
### Get Vault
```
GET /api/secret-provider-configs/{id}
```
### Update Vault
```
PATCH /api/secret-provider-configs/{id}
{
"displayName": "Prod US-East-2",
"config": {
"region": "us-east-2",
"kmsKeyId": "arn:aws:kms:us-east-2:123456789012:key/abcd-..."
}
}
```
`config` is replaced wholesale on update — pass the full provider config
payload, not a partial diff. Status transitions for `gcp_secret_manager` and
`vault` are constrained to `coming_soon` and `disabled` until their runtime
modules ship.
### Disable Vault
```
DELETE /api/secret-provider-configs/{id}
```
Soft-deletes the vault: status flips to `disabled`, `isDefault` clears, and
`disabledAt` is stamped. Disabled vaults remain in `GET` results for audit
purposes but are no longer offered in the secret create/rotate flow.
### Set Default
```
POST /api/secret-provider-configs/{id}/default
```
Marks the target vault as the default for its provider family and clears the
previous default. Returns 422 when the target is `coming_soon` or `disabled`.
### Run Health Check
```
POST /api/secret-provider-configs/{id}/health
```
Runs a provider-specific health probe and persists the result on the vault.
Response shape:
```json
{
"configId": "<uuid>",
"provider": "aws_secrets_manager",
"status": "ready" | "warning" | "error" | "coming_soon" | "disabled",
"message": "Provider vault is ready to handle managed writes",
"details": {
"code": "provider_ready",
"message": "...",
"guidance": ["..."]
},
"checkedAt": "2026-05-06T14:00:00.000Z"
}
```
Health responses never include provider credentials or secret values. For AWS
vaults, `details.guidance` may include missing non-secret env names and the
expected AWS SDK credential source; coming-soon vaults always return
`status: "coming_soon"` with `code: "runtime_locked"` and never call into
provider modules.
### Selecting A Vault When Creating Or Rotating Secrets
`POST /api/companies/{companyId}/secrets` and
`POST /api/secrets/{secretId}/rotate` both accept an optional
`providerConfigId` field that pins the secret to a specific vault. When
omitted (or null), the operation runs through the deployment-level provider
configuration — the same path existing installs already use. The board UI
preselects the company's default vault for the chosen provider before
submitting, so callers should usually send an explicit `providerConfigId`.
Coming-soon and disabled vaults are rejected with a 422; a vault that does not
match the secret's provider is rejected the same way.
```json
POST /api/companies/{companyId}/secrets
{
"name": "prod-stripe-key",
"provider": "aws_secrets_manager",
"providerConfigId": "<vault-uuid>",
"managedMode": "external_reference",
"externalRef": "arn:aws:secretsmanager:us-east-1:123456789012:secret:paperclip/prod/stripe"
}
```
### Response Redaction Rules
Every route in this surface enforces the same redaction contract:
- Secret values are never returned. The board UI never has a "reveal value"
affordance; resolution happens server-side at runtime under a binding.
- Provider credential values are never accepted, stored, returned, logged, or
echoed in error messages. Submitting credential-shaped fields fails
validation with a non-leaking error.
- Activity log entries record vault id, provider, displayName, status, and
isDefault transitions — never `config` payloads or health detail bodies.
## Remote Import From AWS Secrets Manager
Remote import links existing AWS Secrets Manager entries into Paperclip as
`external_reference` secrets. Import stores provider reference metadata only; it
does not copy the remote secret plaintext into Paperclip.
The routes are board-only and company-scoped. `providerConfigId` must point to
a same-company AWS provider vault with status `ready` or `warning`. Disabled,
coming-soon, non-AWS, and cross-company vaults are rejected. Imported secrets
resolve later through the selected vault, so runtime reads still need
`secretsmanager:GetSecretValue` and any required KMS decrypt permission on the
selected external secret.
### Preview Remote Import Candidates
```
POST /api/companies/{companyId}/secrets/remote-import/preview
{
"providerConfigId": "<aws-vault-uuid>",
"query": "stripe",
"nextToken": "opaque-provider-token",
"pageSize": 50
}
```
`query` is optional and is passed to AWS Secrets Manager inventory filtering.
Treat it as non-secret metadata because AWS may record list request parameters
in CloudTrail. `nextToken` is an opaque AWS cursor; callers must pass it back
unchanged and must not synthesize offsets. `pageSize` is optional, defaults to
50 in the UI, and is capped at 100.
Preview uses AWS `ListSecrets` only. It must not call `GetSecretValue` or
`BatchGetSecretValue`, must not request `SecretString`, and must not require KMS
decrypt. The response contains sanitized metadata for display and conflict
decisions:
```json
{
"providerConfigId": "<aws-vault-uuid>",
"provider": "aws_secrets_manager",
"nextToken": null,
"candidates": [
{
"externalRef": "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/stripe",
"remoteName": "prod/stripe",
"name": "prod/stripe",
"key": "prod-stripe",
"providerVersionRef": null,
"providerMetadata": {
"createdDate": "2026-05-06T00:00:00.000Z",
"lastChangedDate": "2026-05-06T00:00:00.000Z",
"hasDescription": true,
"hasKmsKey": true,
"tagCount": 3
},
"status": "ready",
"importable": true,
"conflicts": []
}
]
}
```
Candidate statuses:
- `ready`: the row can be selected for import.
- `duplicate`: a Paperclip secret already links the same canonical provider
reference for the same provider vault.
- `conflict`: the row has a name/key collision or provider guardrail failure.
Conflict types are `exact_reference`, `name`, `key`, and
`provider_guardrail`. AWS refs under Paperclip's own managed namespace are
blocked as external references; use the Paperclip-managed secret flow for those
resources instead.
### Import Selected Remote References
```
POST /api/companies/{companyId}/secrets/remote-import
{
"providerConfigId": "<aws-vault-uuid>",
"secrets": [
{
"externalRef": "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/stripe",
"name": "Stripe production key",
"key": "stripe-production-key",
"description": "Stripe key used by production checkout",
"providerVersionRef": null,
"providerMetadata": {
"createdDate": "2026-05-06T00:00:00.000Z"
}
}
]
}
```
The `secrets` array accepts 1-100 rows. Each row may override the suggested
Paperclip `name`, `key`, optional Paperclip `description`,
`providerVersionRef`, and sanitized `providerMetadata`. Blank descriptions are
stored as `null`; AWS provider descriptions are not copied into Paperclip
descriptions. The backend re-checks duplicate refs and name/key conflicts at
submit time; a stale preview does not bypass those checks.
The import response is row-level:
```json
{
"providerConfigId": "<aws-vault-uuid>",
"provider": "aws_secrets_manager",
"importedCount": 1,
"skippedCount": 1,
"errorCount": 0,
"results": [
{
"externalRef": "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/stripe",
"name": "Stripe production key",
"key": "stripe-production-key",
"status": "imported",
"reason": null,
"secretId": "<paperclip-secret-id>",
"conflicts": []
}
]
}
```
Row statuses:
- `imported`: Paperclip created an active `external_reference` secret and one
metadata-only version row.
- `skipped`: the row had an exact-reference duplicate or name/key conflict.
- `error`: the provider rejected the reference or the row failed validation.
Activity logs for preview/import store aggregate counts, provider id, and vault
id only. They must not store remote secret names, ARNs, descriptions, tags,
plaintext values, provider credentials, or raw AWS error blobs.
## Rotate Secret
```
POST /api/secrets/{secretId}/rotate
{
"value": "sk-ant-new-value..."
}
```
Creates a new version of the secret. Agents referencing `"version": "latest"` automatically get the new value on next heartbeat.
Creates a new version of the secret. Agents referencing `"version": "latest"`
automatically get the new value on next heartbeat. Pin to a specific version
when a bad `latest` rollout would affect many agents at once.
## Using Secrets in Agent Config
@ -52,4 +393,20 @@ Reference secrets in agent adapter config instead of inline values:
}
```
The server resolves and decrypts secret references at runtime, injecting the real value into the agent process environment.
The server resolves and decrypts secret references at runtime, injecting the
real value into the agent process environment. Paperclip's custody guarantees
end at injection: the agent process can read, log, or forward the value, so
treat any secret bound to an agent as exposed to that agent. See the custody
boundaries note in the [secrets deploy guide](/deploy/secrets#custody-boundaries).
## Portability
Company export/import APIs represent agent and project environment requirements
as declarations in the package manifest. Exports omit secret values, secret IDs,
provider references, and encrypted provider material. Use:
```sh
pnpm paperclipai secrets declarations --company-id {companyId}
```
to inspect the declarations that an export would emit before moving a package.