Risk Watchlist — Triage, Notification Pilot, and Policy Runbook

Risk watchlist — triage, notification pilot, and policy runbook

This runbook covers the retraction risk watchlist surface (OPS-RISK-001): the dashboard Risk watchlist route, the guarded triage actions behind it, the notification pilot, and the policy gates that decide when the signal set may grow.

Code lives in apps/dashboard/src/server/functions/risk-watchlist/ (model, detection, API surface) and packages/feature-risk-watchlist/ (dashboard UI). The watchlist store is the AOA database (risk_watchlist table); entries are routed by workspace entity.

Watchlist model

• One open entry per (workspace, publication) is enforced by a partial unique index — a resolved/suppressed entry is never resurrected; fresh evidence creates a new entry.
• States: new → acknowledged → under_review → resolved | suppressed. resolved and suppressed are terminal. No transition leaves a terminal state.
• Severity (impact) and confidence (trust) are independent catalogue ranks — severity is never derived from confidence and detection cannot silently downgrade an entry.
• Critical entries left in new past the escalation window (24h, RISK_WATCHLIST_POLICY.escalationWindowMs) are escalation candidates.
• Every refresh records observation_count (same value re-observed) and flip_count (state changes); last_evidence holds the raw observed/previous values.

v1 signal catalogue (locked — do not extend without §Expansion gate)

Signal	Meaning	Severity	Confidence	Notifies	Watchlisted
retraction_detected	Provider enrichment reports the record retracted	critical	high	yes (pilot)	yes
retraction_flapped	Retraction flag changed more than once	high	medium	no (pilot)	yes
retraction_cleared	Previously retracted record reports not retracted	medium	high	never	yes

The catalogue is a typed exhaustiveness-checked constant (RISK_SIGNAL_CATALOGUE in risk-model.ts); a test locks the shipped set. Adding a kind or a signal family is a deliberate decision under §Expansion gate, never a code default.

Triage procedure

Who: workspace admins/owners (superadmin bypasses). Workspace members are rejected at the API with unauthorized.

1. Open the watchlist — Dashboard → Risk watchlist. Entries are ranked by severity, then freshness. Use the state filter (default Active: new / acknowledged / under_review).
2. Acknowledge (new → acknowledged) when you have seen the signal but cannot review it right now. Critical entries left in new beyond 24h are escalation candidates — if you acknowledge instead of reviewing, say so in the review note.
3. Start review (under_review) before investigating. Every signal requires a visibility-review decision (below) before resolution.
4. Resolve when the investigation concluded. Resolution records the resolver identity and time (resolved_by, resolved_at) and is audited (risk_watchlist.triaged activity).
5. Suppress only for false positives or records that should never surface again (see below). Suppression is a judgment call you own; it hides future noise without deleting history.
6. Use View record to open the consumer API record for the underlying publication.

Illegal transitions (for example resolving a critical entry without a review pass, or any action on a terminal entry) are rejected with illegal_transition and mutated nothing.

Suppression policy

• Resolve = “we investigated, this needed the visibility review, here is the outcome.” Default for genuine retraction signals.
• Suppress = “this entry must not surface as new work again” — use for confirmed false positives (provider misreports a non-retracted record as retracted) and records outside the workspace’s stewardship scope.
• Suppressed entries stay in history (state filter → Suppressed). If the same publication produces another open evidence change, a new entry appears — suppression does not hide fresh evidence, it only prevents the old entry from recurring as open work.
• After suppressing a false positive, check the provider source: repeated false positives for the same provider are watchlist-trust evidence for the §Validation protocol.

Visibility-review decisions

Every retraction signal in the catalogue requires a visibility-review decision (visibilityReviewRequiredFor is true for all v1 signals) — the retraction flag is part of publication visibility, not just metadata hygiene.

• Confirmed retracted → verify the publication’s dashboard visibility state matches the retraction outcome (hidden or annotated per visibility policy) before resolving. Cite the provider source in the review note.
• Cleared upstream → confirm the visibility change mirrors the upstream clearing before resolving retraction_cleared entries. retraction_clearing flips visibility the other direction; treat an unexpected flip as a flap and re-open.
• Flapped → the upstream flag is unstable; hold the record for manual verification and note the providers involved. Do not automate visibility changes from flapped entries.

Notification pilot (OPS-RISK-001 TKT-004)

Notifications are off by default. The pilot exists to prove signal value before any fan-out becomes routine.

• Pilot scope (code, not prose): a signal notifies only if it is catalogue-eligible and simultaneously in the top rank for severity and confidence (RISK_WATCHLIST_NOTIFICATION_PILOT in risk-model.ts). During v1 that admits exactly retraction_detected (critical + high). retraction_flapped (high, medium) and retraction_cleared (medium, high) are watchlist-only.
• Enable path: set RISK_WATCHLIST_NOTIFICATIONS_ENABLED=1 in the dashboard Worker environment (apps/dashboard/wrangler.jsonc vars or the deploy secret store), then deploy. Notifications queue as site email events (risk.retraction.detected) with a per-entry occurrence id.
• Noise threshold (code, not prose): at most RISK_WATCHLIST_POLICY.maxNotificationsPerRefresh (10) notifications are queued per refresh run. A mass-detection event watchlists everything but notifies only the first ten — the rest are silent work, verifiable on the list itself.
• Disable path: set RISK_WATCHLIST_NOTIFICATIONS_ENABLED=0 (or remove the var) and deploy. Watchlist surface, refresh, and triage keep working without notifications.
• Pause review: if the triage team resolves fewer than half of notified entries as actionable, disable notifications and record the finding under §Validation protocol before re-enabling.

Validation protocol & trust gate

The watchlist is trusted when stewards routinely act on it without re-verifying raw provider data. Before the first notification pilot and before any catalogue expansion, run the validation protocol:

1. Sample — pick 10 open entries spanning all three signal kinds after a refresh run (refresh reports counts and reported cohorts per run).
2. Verify against known retraction cases — for each sampled entry, check the publication against the provider source (Crossref/OpenAlex retraction metadata) via the View record link. retraction_detected must show is_retracted: true upstream; retraction_cleared must show a genuine un-retraction; retraction_flapped must show ≥2 upstream state changes.
3. Verify the false-positive path — confirm that a deliberately non-retracted control record produces no entry after a refresh.
4. Record results — log sample size, match count, and mismatches in the stewardship log (workspace setting or the triage review note per entry).
5. Trust gate — the v1 set is trusted when ≥80% of a 10-entry sample verifies clean across the three kinds. Below that, keep notifications off, triage through the watchlist surface only, and file provider-specific false positives as ingestion defects.

Expansion gate

New signal kinds or new signal families (beyond retraction) require, in order:

1. A trusted first watchlist (§Validation protocol passed, recorded in the stewardship log).
2. A written catalogue amendment PR adding the entry to RISK_SIGNAL_CATALOGUE with explicit defaultSeverity, confidence, notificationEligible, and visibilityReviewRequired ranks — exhaustiveness at the type level means nothing ships half-specified.
3. RISK_WATCHLIST_NOTIFICATION_PILOT widened explicitly in the same PR (with test updates), never by default, and only for the new signal if it lands in the top ranks.
4. Rollback preplanned: the new kind must be removable by reverting the PR without orphaning existing entries (old entries keep their state; storage is not kind-coupled).

Rollback path

• Disable notifications — set RISK_WATCHLIST_NOTIFICATIONS_ENABLED=0, deploy. Immediate, reversible, and preserves all watchlist history.
• Narrow notifications to the most trustworthy cases — set RISK_WATCHLIST_NOTIFICATION_PILOT to fewer (or no) severity/confidence ranks and deploy; the code gate enforces it per queued notification. This stops the notification scope from growing without touching entry creation or history.
• Stopping signal creation requires a code change — catalogue eligibility governs notifications, not entry creation; detection creates a watchlist entry for any retracted observation from a supported kind. To stop creating entries for a kind, gate it in the detection pipeline (a deliberate PR, same review bar as the expansion gate). Existing entries and their history remain in the AOA store untouched.
• Full stop — keep the feature deployed but stop refreshes (do not action the refresh endpoint from operator tooling); the dashboard list remains read-only history. No data deletion is required at any point.