Bulk Curation — Runbook (Guarded Bulk Delete, OPS-CURATE-001)

Bulk curation — runbook

This runbook covers the guarded bulk-delete surface for publications, works, and projects (bulkPublicationAction, bulkWorkAction, bulkProjectAction). It is the single place where operator flow, rejection semantics, telemetry language, and the rollout gate stay aligned.

Contract code: apps/dashboard/src/server/functions/bulk-curation/ (contract.ts, curation-resolvers.ts, gate.ts). Schemas: packages/schemas/src/index.ts (BulkPublicationActionSchema, BulkWorkActionSchema, BulkProjectActionSchema).

Catalogue and limits

• Action catalogue: delete only. Broader actions must pass the decision gate below and be added to BULK_CURATION_ALLOWED_ACTIONS plus all three Zod schemas deliberately.
• Hard limits (BULK_DELETE_LIMITS in contract.ts): at most 100 ids per command, preview examples capped at 5, preview TTL 10 minutes.
• Permission model: workspace_admin only. Members are rejected unauthorized before any scope resolution or dispatch; the rejection is recorded as an activity event.

Operator flow (the only supported sequence)

1. Preview (mode: "preview"): resolves the workspace-visible scope through the owning domain read path (DOI-sharded D1 for works/projects; publication query service for publications). Mutates nothing. Returns preview_id, selection_digest, preview_issued_at, affected_count, out_of_scope, and up to 5 examples.
2. Confirm in the client dialog (naming the affected count + out-of-scope warning). The client sends confirmation: "CONFIRM" — hardcoded client-side, not user-typed.
3. Execute (mode: "execute" + confirmation fields): execution re-resolves scope through the same resolver the preview used, so the executed set is always a subset of what was previewed; drift is reported, never silently dispatched.

Execute gate order (see bulkWorkActionMutation / bulkProjectActionMutation / bulkPublicationActionMutation):

preview_id presence → confirmation + digest + freshness validation → scope re-resolution → KV preview-claim (duplicate prevention) → per-item dispatch.

Rejection codes and what they mean

Code	Meaning	Operator action
unauthorized	caller is not a workspace admin of the target workspace	nothing dispatched; verify role
empty_scope	zero ids in selection	nothing to do
scope_too_large	> 100 ids	split the selection into batches
preview_expired	preview older than TTL	re-issue preview, re-confirm
confirmation_mismatched	supplied selection does not hash to the previewed digest	re-issue preview for the current selection
duplicate_operation	this preview_id was already executed	re-issue a fresh preview; the claim is sticky for the TTL
out_of_scope	every requested id is invisible to this workspace	check the selection and workspace

Nothing is deleted on any rejection; rejected events carry bulk_delete.rejected in the activity log with the rejection_code.

Failure modes, recovery, and non-atomicity limits

• No cross-item transaction. Each item dispatch is an independent write (works/projects: DOI-sharded D1 batch deleting relationship rows + entity row; publications: write-service /v1/delete). There is no atomic rollback across items. A failed or abandoned run leaves earlier successes intact and a failed cohort behind in the run envelope (items list).
• Partial/failure cohorts are honest. counts.failed / counts.success / counts.skipped always describe exactly what happened; items that fail dispatch are returned with outcome: "failed" and reason: "dispatch_failed". Recovery = re-run a new preview → confirm → execute cycle over the failed cohort only.
• Duplicate prevention is best-effort. The KV claim degrades open if the KV read/write errors (a captureMessage warning is emitted) — the delete itself remains idempotent per item (second delete of a missing row resolves as skipped, not an error), so re-execution after a degraded-claim incident cannot double-delete anything.
• Scope drift is expected and safe. Items deleted between preview and execute come back as skipped with reason: "not_found_in_workspace" and are never dispatched.
• Selection drift fails closed. A client that edits the selection after preview gets confirmation_mismatched; the only path to execute is re-preview + re-confirm.

Telemetry

• Route metrics (Cloudflare Observability → dashboard Worker logs): every completion logs route publications.bulk-delete / works.bulk-delete / projects.bulk-delete with outcome (success / partial / failed / preview-mode success / unauthorized / rejected), target, mode, and (executes only) counts + operation_id — see withMutationTelemetry wiring in each domain’s index.ts.
• GlitchTip: unexpected per-item dispatch failures and thrown handler errors carry tags.source: "bulk-curation" (dispatch items) or source: "ingestion-ops" (wrapper), plus entity_id / operation_id extras.
• Activity events (AOA D1, surfaced in dashboard Activity): *.bulk_delete.rejected with rejection_code; *.bulk_delete.executed summary with counts; work.deleted / project.deleted / publication.deleted per item with bulk_operation_id for correlation.

Validation

The rehearsal suite (src/server/functions/bulk-curation/rehearsal.test.ts) drives the real resolvers, write plans, and dispatchers against stateful binding fakes and asserts:

• preview/execute fidelity for all three domains (including relationship-row cleanup),
• drift between preview and execute surfaces as skipped/executed subsets (never unexplained),
• cross-workspace selections can never be deleted,
• failed dispatches report honestly and keep the store intact,
• member denials, empty scopes, and oversized scopes are rejected before resolution/dispatch,
• repeat execution of the same confirmed preview is impossible while the KV claim holds.

Run it with:

pnpm -F @apps/dashboard exec vitest run src/server/functions/bulk-curation/rehearsal.test.ts

Broader curation — decision gate

Expansion beyond delete stays disabled by default. The gate is implemented in bulk-curation/gate.ts (evaluateDeletePilotGate) and evaluated with metrics gathered from a pilot window with internal admins:

Threshold	Value	Rationale
min_previewed	≥ 10 previewed items	rates are meaningless on smaller samples
preview_drift	≤ 5% previewed items unaccounted for at execute	every previewed item must resolve to executed or skipped
success_rate	≥ 95% of validated executions succeed	delete reliability floor

A pilot passes when evaluateDeletePilotGate(metrics).pass === true and no unauthorized / scope_too_large / confirmation_mismatched bypass regressions are open. Broader actions (restore, visibility, merge, export, …) require all of:

1. the pilot threshold met on real (not rehearsed) traffic,
2. a catalogue change in gate.ts + schema changes for every new action,
3. the rehearsal suite extended to cover the new action’s guard-first rejection paths,
4. this runbook updated with the new rejection codes and recovery story.

Until then, catalogueSupportsAction(action) is true only for delete, and the Zod schemas reject every other action value at the boundary.