Ingestion Observability Console — Runbook & Alert Reference

Ingestion observability console — runbook & alert reference

This runbook covers the ingestion observability v1 surface: the Activity → Queue Status card (signal contract + drill-downs) and the guarded remediation actions behind it. It is the single place where panel language, alert language, and first actions are kept aligned.

The surface extends the existing dashboard Activity route. There is no dedicated observability worker or store; direct D1 job state is authoritative, activity-derived and Cloudflare-derived values are labelled as inferred/proxy.

Signal groups, sources, and freshness budgets

Every value in the signal contract carries provenance (direct / inferred / proxy / unavailable), availability (available / partial / unavailable), freshness (fresh / stale / unknown), a source label, and an observation time.

Panel	Signals	Source	Provenance	Stale when
Workload state	total / running / queued jobs	transactional_d1.ingestion_jobs	direct	never (live read)
Workload state	publication/work resync running & waiting	aoa_d1.activity_events	inferred	newest event > 60 min old
Queue age & throughput	ages, p50/p95, throughput/min	transactional_d1.ingestion_jobs	direct	never (live read)
Failures, retries & stuck	counts, rates, stuck list	transactional_d1.ingestion_jobs	direct	never (live read)
Provider & rate limits	rate-limit timeouts, failure events, top provider	aoa_d1.activity_events	inferred	newest event > 60 min old
Provider & rate limits	backlog, concurrency, lag, retries	cloudflare.graphql.queue_metrics	proxy	bucket older than 10 min
Shard partiality	configured / readable / failed shards	transactional_d1.shard_reads	direct	never (live read)
Recovery history	nudge history	aoa_d1.activity_events	direct	newest event > 60 min old
Recovery history	automatic orphan recovery	dashboard.activity.legacy_recovery	direct	never
Drill-downs	failure cohorts 24h, recent failed jobs, retrying jobs, shard status	transactional_d1.ingestion_jobs	direct	never (live read)
Drill-downs	provider breakdown (15m / 1h windows)	aoa_d1.activity_events	inferred	newest event > 60 min old
Health · Data freshness	researcher sync age (threshold 30d), publication fetch age (90d), enrichment pending	transactional_d1.people+publications (per-shard fan-out)	direct	shard read failures → partial/unknown
Health · Review & visibility	pending work reviews, hidden publications, author-match attention	transactional_d1.work_review_items+publications (per-shard fan-out)	direct	shard read failures → partial/unknown
Health · Workspace readiness	active integration installations, configured email/Discord channels	global_d1.integrations_installations+entities (DB_0)	direct	configuration read failure → unknown

Freshness budgets are defined in apps/dashboard/src/server/functions/activity/transformers.ts (ACTIVITY_SOURCE_STALENESS_THRESHOLD_MS, CLOUDFLARE_SOURCE_STALENESS_THRESHOLD_MS).

Degraded-state semantics

• Partial — readable values are shown but some transactional shards could not be read. Direct totals may undercount. Do not treat zero as “healthy” on a partial panel.
• Unavailable — the source could not be read at all. The panel shows the reason string; provider panels are hidden behind an explicit unavailable state rather than zeros.
• Stale — the source is readable but its newest observation exceeds the freshness budget. Values stay visible with reduced confidence and a stale marker; the contract lists offending sources in signal_contract.stale_sources (also exposed top-level as stale_sources).
• Empty — the source is readable and genuinely has no rows (for example no failed jobs in 24h). This is a healthy zero and is distinct from partial/unavailable.
• Newly initialized — a workspace with no members and no publications has no freshness signals; the freshness bucket reports unknown with a freshness.no_data explanation rather than a false healthy state.

Workspace health signal catalogue (health route)

getWorkspaceHealth returns the four buckets. Thresholds live in apps/dashboard/src/server/functions/workspace-health/transformers.ts:

Evidence code	Severity	Threshold / meaning	Owning surface
freshness.researcher_sync_stale	warning	researcher orcid_last_synced_at older than 30 days (or never synced)	People
freshness.publication_stale	warning	publication last_fetched_at older than 90 days	Publications
freshness.publication_enrichment_pending	info	enrichment_status != 'complete'	Publications
freshness.no_data	info	newly initialized workspace (0 members, 0 publications)	—
freshness.checked	info	healthy baseline: what was checked and against which thresholds	—
review_visibility.pending_work_reviews	warning	orcid_work_review_items with review_status = 'pending'	Works review (unverified)
review_visibility.hidden_publications	warning	publications with visible = 0 and not deleted	Publications
review_visibility.author_matches	info	publications with author_match_state = 'needs_attention'	Publications
review_visibility.checked	info	healthy baseline: nothing pending, hidden, or unattended	—
readiness.integrations_none_active	warning	no integrations_installations row with status = 'active' (archived excluded)	Integrations
readiness.notifications_unconfigured	warning	no email/Discord channel in the workspace’s notification destinations	Workspace settings
readiness.checked	info	healthy baseline: installation active and channels configured (setup status only, never third-party provider health)	—

All counts are direct D1 aggregates read per transactional shard and summed. Failed shard reads downgrade affected buckets to partial (some shards missing) or unknown (no shards readable) with an explanation — never to a healthy zero.

Workspace health route metrics

getWorkspaceHealth logs structured route metrics (dev-gated like other reads) with per-bucket state, evidence counts, and failed-shard counts, so health-query latency and missing-source behavior are visible in Cloudflare Observability.

Alert mapping (GlitchTip)

Alert rules should key on these exact capture strings and tags. All are emitted by the dashboard server runtime.

GlitchTip message	Level	Tags	Meaning	First action
Activity queue status read with unreadable transactional shards	warning	source:activity	≥1 transactional shard failed during the queue-status read	Check D1 shard health; expect partial_data + drill-down panels to show partial totals
Activity queue status read with unavailable activity event source	warning	source:activity	The activity event store could not be read	Check AOA database availability; provider/resync panels will be unavailable
Ingestion force-fail rejected: job not found or not active	warning	source:ingestion-ops	An operator tried to force-fail a job that already finished or left the workspace	Usually stale UI; no action if transient
Ingestion requeue rejected: membership is external or outside	warning	source:ingestion-ops	Requeue attempted for an ORCID without an active membership	Confirm membership state before re-ingesting
Ingestion requeue failed to persist job	error	source:ingestion-ops	Insert of the requeued job failed — no queue dispatch happened	Investigate immediately; the ORCID is NOT queued
Reprocessing command rejected: ... (rejection_code in message + metadata)	warning	source:reprocessing	A bounded reprocessing command was rejected (limits, confirmation, duplicate, gate)	Usually operator error or a stale preview; a spike in duplicate_command indicates a retry loop
Reprocessing scope resolution failed (via captureException)	error	source:reprocessing	D1 scope reads failed after confirmation and KV claims — nothing was dispatched	Check shard health, then re-request a fresh preview; the command id may be reused
dispatch_failed cohorts (executed-event counts.failed > 0 via captureException)	error	source:reprocessing	Queue sendBatch failed after rows were transitioned — messages DID NOT reach the queue	Verify queue availability; affected rows are already queued in D1, so a re-run is safe (no second active job)
activity event write failed / activity_events table missing — writes skipped	warning	source:activity	Activity history is degraded (best-effort writes skipped)	Expect partial_data/unavailable activity panels; fix the writer path

Unexpected exceptions in any mutation (nudgeActivityQueue, cancelAllJobs, forceFailJob, requeueIngestionOrcid) and in the queue-status read are captured with captureException (tagged source:ingestion-ops for mutations with extra.route, source:activity for reads) and rethrown, so existing error responses are unchanged.

Cloudflare Observability (structured route metrics)

Mutations always log a completion metric (operator-triggered, low volume); the polled read path logs verbose metrics in dev only:

Route	Gating	outcome vocabulary	Extras
getActivityQueueStatus	dev-only verbose	(log on completion)	failed_shards, activity_source_freshness, stale_sources, failure_cohorts
nudgeActivityQueue	always	accepted / unauthorized	matched, failed_stale_jobs, requeued_jobs, skipped_requeue_due_to_active_job
cancelAllJobs	always	accepted / unauthorized	cancelled_count
forceFailJob	always	accepted / rejected / unauthorized	job_id, orcid_id
requeueIngestionOrcid	always	accepted / skipped / rejected / failed / unauthorized	orcid_id, queued, skipped_reason
requestReprocessing	always	accepted / skipped / rejected / failed / unauthorized	command_type, mode, code, counts (executed), rejection_code

outcome: failed on requeueIngestionOrcid always pairs with the error-level GlitchTip message above. outcome: skipped is expected UI-staleness traffic, not an incident. For requestReprocessing, outcome: failed (ledger failed > 0 with requeued === 0) always pairs with a dispatch_failed capture; outcome: rejected covers both operator errors and the configured launch gate (execute_disabled).

Activity history (attributed events)

Every remediation outcome writes a workspace activity event (best-effort, attributed to the acting operator): ingestion.queue.nudged, ingestion.queue.cancelled, ingestion.queue.auto_recovered (legacy autonomous recovery), ingestion.job.force_failed, ingestion.job.requeued. Authorization failures (403) are not written to workspace history; they appear only in route metrics with outcome: unauthorized.

Incident-class playbooks

No active consumers (queue draining stalls)

1. Panel: Queue age & throughput — no_active_consumers waiting reason; queued jobs > 0, running = 0.
2. Confirm consumer deployments (ingestion orchestrator/process workers) are healthy.
3. Superadmin: use Recover now (bounded clear+requeue) from the waiting-reason panel.

Retry backoff pressure

1. Panels: Failures, retries & stuck (retry rate, retrying jobs) and Retrying jobs drill-down (highest attempt counts first, with shard).
2. Identify the dominant Failure cohort (24h) and check Recent failed jobs for the error message and shard.
3. If a specific cohort dominates, fix the underlying cause before nudging; nudging requeues whole ORCIDs, not error classes.

Provider rate-limit impact

1. Panel: Provider & rate-limit impact + Provider breakdown drill-down (15m / 1h windows per provider, last-seen timestamps).
2. If the activity source is stale/unavailable, provider panels say so — do not infer “no provider impact” from an empty unreliable panel.
3. Mitigate upstream (provider quotas) rather than requeueing; rate-limited work retries itself.

Shard partiality or hotspots

1. Panel: Shard partiality (readable vs failed) and Shard status drill-down (per-shard queued/processing/failed-today; unreadable shards listed).
2. A failing shard marks direct panels partial — compare cohorts across readable shards only.
3. Check D1 shard health for the failed shard index; totals are incomplete until it recovers.

Stuck jobs

1. Panel: Stuck jobs list (waiting age, attempts, shard) — top of page for superadmin actions (copy id, force fail, requeue ORCID).
2. Force-fail is workspace-scoped and bounded; requeue skips when an active job already exists.

Validation record

• Signal calculations and degraded-state rendering are regression-tested at three seams (transformer unit tests, mocked read-path tests, component tests) covering reliable, empty, partial, unavailable, and stale inputs, including out-of-window (future-dated) provider events.
• Mutation telemetry outcomes are covered by queue.test.ts and remediations.test.ts (accepted / skipped / rejected / failed / unauthorized / captured-exception paths).
• Production validation (controlled failure drill against staging, then pilot) happens at rollout; displayed values must be compared against D1 job state and known activity events. Cloudflare aggregate metrics are checked only against their declared minute-bucket granularity.

Reprocessing command model (OPS-ING-001)

requestReprocessing is the single bounded entry point for reprocessing commands. All three families support preview; every family also supports guarded execution (bounded identifier-driven controls only — there is deliberately no global reprocessing or generic cohort engine).

• Approved families (anything else is unsupported_command):
  • stale_failed_replay — threshold (60s–86400s, default 900) + limit (≤ 500, default 200).
  • orcid_sync — 1–100 explicit ORCIDs, skip_existing defaults to true.
  • resync — publication (DOI-normalized ids, 1–50) or work target (ids resolve against the workspace’s works table and must be ORCID-sourced with a put-code, matching the single-item work-resync control).
• Hard limits (REPROCESSING_LIMITS): oversized selections are rejected with oversized_selection and the offending limit — never silently truncated.
• Identity: every command carries a client command_id; the server returns a fresh operation_id echoed in the result and the attributed reprocessing.requested / reprocessing.rejected activity events.
• Confirmation & expiry: previews need no confirmation. Execution of any family requires the literal CONFIRM plus a preview issued within 10 minutes (confirmation_required, confirmation_mismatched, preview_missing, preview_expired otherwise) and claims the client command_id in KV for 10 minutes — duplicate executes are rejected explicitly with duplicate_command (KV unavailability degrades the guard instead of blocking dispatch).
• Shared eligibility predicates (reprocessing/bounded-controls.ts): ORCID queue-entry eligibility (active-job suppression + external/outside membership screening), entity-scoped publication existence, and the works/put-code resync rule are the same helpers used by preview scope and dispatch, so a confirmed preview cannot diverge from execution-time checks.
• Execution ledger (any family + mode: "execute"): all families execute through the same ledger shape — requested / accepted / requeued / skipped / rejected / failed counts plus bounded cohort samples ({ identifier, reason }, ≤ 20 rows each).
  • stale_failed_replay: dispatch re-selects the eligible set and re-qualifies each row with a conditional transition before sending (see below for reasons).
  • orcid_sync: dispatch screens membership, suppresses identifiers with an active job (active_job_exists — there is never a second active job per ORCID; the partial unique index is the final guard), inserts queued jobs, and dispatches ≤ 100-message chunks. skip_existing stays processor-side in the job message (existing batch semantics).
  • resync: each DOI is re-checked at dispatch — publications must exist in the workspace (not_found_in_workspace otherwise); works must additionally be ORCID-sourced with a put-code (not_orcid_sourced otherwise). Dispatch failures are dispatch_failed and are captured to GlitchTip.
  • In every family requeued only counts messages actually handed to INGESTION_QUEUE, and membership-restricted identifiers are rejected with membership_external_or_outside.
• Replay-specific ledger reasons: no_longer_eligible (raced or duplicate), active_job_exists (unique-index guard), transition_failed, membership_external_or_outside (rejected), dispatch_failed (queue sendBatch failure after the row transitioned; captured to GlitchTip).
• Monitoring: every completion logs route metrics (requestReprocessing, outcome accepted/rejected/failed/skipped/unauthorized); accepted commands emit one attributed reprocessing.executed activity event (command id, operation id, counts, threshold/trimmings). Active runs and queue lag remain visible through the ingestion queue-status surface; failure cohorts, partial results, and cancellation semantics travel in the ledger and the event metadata. Activity logging is best-effort and can never block or roll back dispatch; scope resolution failures return scope_unavailable (503) before any row is touched.
• Rejection codes: unsupported_command, invalid_command, oversized_selection, invalid_identifier, confirmation_required, confirmation_mismatched, preview_missing, preview_expired, duplicate_command, scope_unavailable, forbidden. All rejections dispatch no queue work and emit an attributed rejection event. Authorization: only the workspace’s workspace_admin/owner roles (or a superadmin) may run commands; members, anonymous callers, and admins of other workspaces get forbidden.
• Telemetry: dispatch failures and scope-resolution failures reach GlitchTip (source:reprocessing); preview latency and outcomes are visible in route metrics.

Reprocessing rehearsal, alerts, and rollout (OPS-ING-001 TKT-004)

Rehearsal record — controlled scenarios (codified as regression tests)

The controlled rehearsal ships as reprocessing/rehearsal.test.ts: every scenario runs the real dispatch machinery against stateful in-memory shards (durable jobs table, partial unique index emulation, membership rows), so CI rehearses the same duplicate-suppression behavior production enforces.

#	Controlled scenario	Expected rehearsal outcome	Tests
R1	Bounded ORCID backfill, then a second identical run	First run: 1 accepted, 1 skipped (active_job_exists), 1 rejected (membership); second run: 0 accepted, 0 new rows, no new queue messages	rehearsal.test.ts (“second run creates no duplicate active work”)
R2	Failed-job replay (stale/failed selection + membership block)	2 stale jobs dispatched, 1 external block rejected; after jobs complete, re-selection matches 0 — recovery converges	rehearsal.test.ts (“replays stale and failed jobs”)
R3	Resync of missing/unresyncable identifiers	Only workspace-resident, put-code-bearing targets reach the queue; skips carry not_found_in_workspace / not_orcid_sourced; no ingestion rows created	rehearsal.test.ts (“bounded resync rehearsal”)
R4	Queue sendBatch failure mid-run (150 messages, 100+50 chunks)	Ledger stays honest: requeued: 100, failed: 50 (dispatch_failed), captureException fired; rows already queued in D1 make a re-run safe	rehearsal.test.ts (“ledger honest when queue sends fail”)
R5	Fully failed dispatch	requeued: 0, failed: 1, captureException tagged source:reprocessing	rehearsal.test.ts (“reports a fully failed dispatch”)
R6	Duplicate execute (same command_id inside the KV TTL)	duplicate_command rejection, zero dispatch calls, across families	index.test.ts
R7	Launch gate disabled (REPROCESSING_EXECUTE_ENABLED=false)	execute_disabled rejection before confirmation; previews still issued	index.test.ts

Staging rehearsal (at pilot time, before production enablement): run one replay and one ORCID backfill from a pilot workspace with the Activity queue-status panel open; compare the executed event’s counts against D1 job rows and the queue depth drop. Record the comparison here.

Alert thresholds — runaway, stalled, partial, and failed runs

Alerting keys entirely on signals that already exist: the reprocessing.executed activity event (counts + command id + threshold), route metrics for requestReprocessing, and the GlitchTip captures in the alert-mapping table. There is deliberately no separate reprocessing metrics pipeline.

Signal (per workspace / entity)	Threshold (initial)	Reading	First action
reprocessing.executed events with counts.failed > 0	any occurrence → error alert	Queue sends failed after row transitions; work is queued in D1 but NOT on the queue	Check queue availability before any other action; re-runs are safe (single-active-job invariant)
reprocessing.executed with requeued << requested (large skipped/rejected cohort)	skipped + rejected > 50% of requested in one command	Partial completion — usually correct-but-surprising (membership or missing identifiers)	Read the skipped/rejected cohort samples in the ledger before re-requesting
reprocessing.executed per entity	> 10 executions per hour per workspace, or > 3 in a 10-minute window	Possible runaway operator loop or retry script	Check the actor attribution; talk to the operator before enabling anything wider
reprocessing.rejected with rejection_code: scope_unavailable	> 3 in 15 minutes	Shard reads degrading at execution; nothing dispatched, operators blocked	Shard health first; execution stays blocked until scope reads recover
reprocessing.rejected with rejection_code: duplicate_command	> 20 in 10 minutes	A client or script is retrying an already-executed command id	Find the retry source; dedupe is working as designed
Stalled run	reprocessing.executed with status: failure and counts.requeued === 0 and counts.requested > 0	Everything skipped or failed — no requeue progress	Inspect the skip/reject cohorts; if transition_failed dominates, check shard write health
Stalled commands (launched but silent)	claimed command_id in KV with no corresponding executed/rejected event within 10 min	Handler died between claim and dispatch (KV entry self-expires in 600s)	Check dashboard runtime errors; claim expiry means a retry with the same command id is safe

Operator workflow

1. Preview (mode: "preview"): read requested/estimated counts, the skipped cohorts, and the warnings. The plan expires in 10 minutes and dispatches nothing.
2. Approval: preview + command id are the approval artifact (what, whom, when, estimates). No execution without a preview fresh enough to confirm against.
3. Launch (mode: "execute", confirmation: "CONFIRM", preview_issued_at from the approved preview): the command id is claimed in KV for 10 minutes; duplicate executes are rejected with duplicate_command.
4. Monitoring: the reprocessing.executed event’s counts, the bounded cohort samples, the Activity queue-status surface (active runs / lag), and route metrics. Partial completion is expected to be visible in counts — never inferred.
5. Cancellation: abandoning before confirmation dispatches nothing. Execution cancellation prevents new dispatch only; work already processed is not rolled back (this is stated in every preview’s cancellation copy).
6. Manual fallback boundary: the controls intentionally cover only identifier-bounded families. Anything outside them (unmatched identifiers, cross-workspace cohorts, unbounded enrichments) stays a manual, ticketed operations task on the existing scripts — the reprocessing surface is not a generic cohort engine.

Rollout decision

• Included at launch: the three bounded families (stale_failed_replay, orcid_sync, resync) in preview and confirmed-execute, workspace-admin/owner/superadmin authorization, per-command hard limits, the KV duplicate-command ledger, and the launch gate below.
• Excluded, explicitly: global automatic reprocessing — no automatic requeue on enrichment-rule changes, no generic cohort engine, no background sweeps of all workspaces (spec non-goal). The only retained autonomous recovery is the existing queue auto-recovery (ingestion.queue.auto_recovered): cooldown-gated requeue of never-delivered orphaned queued jobs, bounded by threshold and limit, emitting an attributed warning event. It predates this surface and is not a reprocessing command.
• Expansion gate: new command families or larger per-command limits land only after the pilot records successful rehearsals here, alert thresholds stay clean through a pilot window, and duplicate suppression is observed in production data (executed-event skipped cohorts matching D1 active-row state on a sample).
• Rollback lever: set REPROCESSING_EXECUTE_ENABLED: "false" in the dashboard Worker vars. Execute requests then reject with execute_disabled (503) before confirmation or any D1/KV writes; previews keep working and run history (activity events, route metrics) is fully preserved. There is no code rollback needed to disable launch capabilities.
• Emergency recovery boundary: if the surface must be bypassed entirely (e.g. a broad incident the commands cannot express), return to the manual scripts as documented in the incident playbooks above — one-off, ticketed, and never automated.

Validation record (OPS-ING-001 TKT-004)

• Authorization, idempotency, scope limits, and duplicate suppression are regression-tested at the command surface (index.test.ts: admin/member/superadmin/anonymous/other-workspace authorization, duplicate_command, oversized selections, confirmation/expiry paths) and at the dispatch seam (replay.test.ts, bounded-controls.test.ts, rehearsal.test.ts).
• Rehearsal outcome: 7 controlled scenarios pass against the real dispatchers with stateful shard fakes (see the rehearsal record table).
• Production validation runs at pilot: compare executed-event counts against D1 job state for the pilot scenarios and record the comparison here.

Workspace health buckets — operator guidance

The workspace health page (/{locale}/workspace-health, admin-gated) composes four independent buckets. There is no overall score by design; judge each bucket on its own evidence.

Bucket	What it answers	Healthy means	First action when not healthy
Ingestion reliability	Is ingestion work progressing?	Checked evidence names running/queued/completed counts	Follow the evidence link into the Activity queue status; use stuck-job drill-downs before nudging
Data freshness	Is trust-sensitive data inside declared expectations?	Researchers within 30d sync, publications within 90d fetch	Use the People/Publications links; check the threshold in the evidence detail first
Review & visibility	What needs curator attention?	Nothing pending, hidden, or awaiting author-match	Use the works-review / publications links; clear the pending cohort before hidden pubs
Workspace readiness	Is configuration complete?	Active installation + at least one email/Discord channel	Use the Integrations / Workspace settings links; configuration gaps are setup work, not incidents

Degraded response: a bucket in partial or unknown must not be read as healthy — trust the explanation text and the badge, and prefer the owning surface for real counts until the source recovers. A failed queue snapshot degrades only the ingestion bucket; freshness, review, and readiness remain usable.

Validation record (OPS-HEALTH-001 TKT-004)

• Controlled source failures are regression-tested at the composition seam: queue-snapshot failure (ingestion → unknown, other buckets usable), total and partial shard failures (freshness/review → unknown/partial), and readiness read failure (readiness → unknown).
• Permission boundaries are regression-tested: workspace admins and owners allowed; members rejected with no source reads; admin roles in other workspaces grant nothing; superadmin requests honor the authorized requested workspace (ADR-0016) with no cross-workspace leakage.
• No-score behavior is asserted on both healthy and fully degraded payloads.
• Sampled production validation (comparing displayed counts against direct D1 queries for a few representative workspaces) runs at pilot time; the queries are the same direct aggregates the buckets display, so drift should be zero by construction — record results here during pilot.

Rollout decision

• Pilot scope: internal workspace admins only (the read is already admin-gated server-side).
• Pilot question: is the page preferred over the manual cross-screen path (People → Publications → Works review → Activity → Settings) for the covered cases? Capture feedback against the four buckets.
• Trim criteria: a card that no pilot admin used during real maintenance, or whose recommendation was wrong more than it was right, is removed or collapsed before broader access.
• Rollback: restrict to the pilot group or hide the affected bucket’s card while its signal is repaired; per-source isolation means one bucket can be hidden without losing the page.

Pilot rollout scope

• The console ships to platform operators only (superadmin-gated remediations; the read surface is workspace-scoped).
• Low-signal panel candidates to re-evaluate with pilot feedback before broader access: Cloudflare concurrency tile (often zero), automatic-recovery tile when never triggered, and the 1h provider windows if 15m proves sufficient. Removal criterion: a panel that no pilot operator used during an incident, or that never changed a decision, is removed or collapsed.
• Rollback trigger: console data materially misleading without a clear degraded-state warning, or unacceptable read cost/latency — disable the UI section first, aggregation slices second.