Recurring ORCID auto-sync runbook

Recurring ORCID auto-sync runbook

This runbook covers workspace-owned recurring ORCID schedules. The feature schedules an existing record sync; it does not create a second provider pipeline, promise immediate completion, or introduce researcher self-service.

Contract and authorization

• A schedule is unique per (entity_id, person_orcid) and is stored in global DB_0 as orcid_sync_schedules. Person records and orcid_sync_jobs remain on the ORCID-routed transactional shard.
• The first release offers fixed Daily and Weekly cadences. Daily uses the existing daily_delta record trigger; weekly uses weekly_full. New enabled schedules are due on the next scheduler tick.
• Workspace owners, workspace admins, and superadmins can inspect, enable, change cadence, pause, and resume. The server derives workspace scope from the authenticated session; clients cannot select a different workspace.
• An active internal (cesam) workspace membership is required to enable or resume a schedule. External or ended memberships produce a recorded skipped window instead of a queue request.

Rollout

1. Apply migrations/drizzle/0048_orcid_sync_schedules.sql to global DB_0 with bash scripts/db/apply-d1-migrations.sh --remote --yes after reviewing the target list.
2. Deploy the dashboard and worker-ingestion-orchestrator together. The orchestrator’s hourly 0 * * * * cron polls due schedules; the existing 0 1 * * * legacy bulk delta remains separate during this rollout.
3. Enable one internal workspace/researcher from the person detail page. Confirm the schedule shows Queued, then Syncing, and finally a last scheduled success and a future next run.
4. Add one intentionally paused schedule and one schedule with a non-syncable membership in a controlled workspace. Confirm the UI reports Paused and Skipped, respectively.
5. Capture the schedule status, worker logs, activity event, and ORCID job row for the rollout evidence. Expand in small waves; keep the number of enabled schedules capped while monitoring provider rate limits and queue lag.

Observability and troubleshooting

The scheduler emits structured Cloudflare logs and activity events. Dashboard schedule changes emit orcid.sync.schedule.updated with the workspace, ORCID, schedule ID, cadence, and action:

• orcid_sync_recurring_schedule_tick — inspected, claimed, enqueued, skipped, and failed counts.
• orcid_sync_schedule_enqueued / orcid.sync.schedule.enqueued — schedule and job correlation.
• orcid_sync_schedule_skipped / orcid.sync.schedule.skipped — active-job and membership skip reasons.
• orcid.sync.schedule.completed and orcid.sync.schedule.failed — worker outcome and job correlation.

GlitchTip captures scheduler/database/queue exceptions through the existing worker monitoring seam. Cloudflare Worker Observability logs and traces remain enabled by the worker configuration.

• Due or delayed: check the hourly cron invocation, DB_0 migration state, and the worker inspected/claimed counts. A due row with no claim may be paused, leased, or in a concurrent active window.
• Skipped: inspect last_skip_reason. active_job:<id> and active_job_race mean another same-workspace/scope job won; workspace_membership_not_syncable means membership must be restored before the next window.
• Failed: inspect latest_failure, consecutive_failures, and the matching orcid_sync_jobs row on the ORCID shard. Enqueue failures back off at 5 minutes, 15 minutes, and 60 minutes; after the bounded retries the schedule waits for its normal cadence.
• Stuck queued/processing: verify queue consumer health and the lease/job IDs. Do not insert a replacement row manually; an expired schedule lease terminalizes the abandoned queued/processing job and lets the next hourly tick reclaim the window. Scheduled queue processing also rechecks internal membership immediately before the provider request.

Pause and recovery

Pause an individual schedule from the person detail page before incident response or provider maintenance. Pausing prevents future claims but does not cancel a request already accepted by the queue. Resume after the incident; the schedule becomes due and the next run is visible in the panel.

For a broad incident, disable the recurring cron in the orchestrator deployment or deploy the previous worker version. Keep orcid_sync_schedules and its status state intact, and use manual ORCID sync for urgent records. Re-enable the hourly trigger only after queue lag, provider limits, and failure rate are normal.

To roll back the schema after the feature is disabled, preserve an export if operational review requires it, then execute migrations/rollback/0048_orcid_sync_schedules.sql on DB_0. This removes only the schedule table and its indexes; it does not remove researcher data or sync jobs. Re-apply migration 0048 before deploying schedule-aware dashboard or worker code again.