incidental

A temporal incident-management system (CLI + API) built as long-term memory for a Kubernetes operator agent (KAOS). Not primarily a human ticketing tool: the primary reader is a machine that re-derives its beliefs on every invocation, so provenance, confidence, and "what did we believe at time T" are first-class.

The full specification lives in docs/01-design-spec.md.

Design in one paragraph

Incidents are event-sourced: an incident is an append-only stream of immutable events (observations, transitions, hypothesis revisions, evidence, actions); current state is a projection. Every event is bitemporal (occurred_at valid time, recorded_at transaction time), so as-of queries reconstruct an incident exactly as the store knew it at any past moment. Identity is a signature (resource kind / namespace / error class / alert name): re-observing a signature appends to the open incident (a pod crashlooping 400 times is one incident), and re-observing a closed lineage opens a numbered recurrence linked with a typed recurrence_of edge — recurrence count per signature is the escalation signal. Hypotheses carry an earned confidence level: probable/confirmed are rejected unless ruled-out alternatives are recorded. Mutations mirror the agent's authority split: executed actions are logged facts, proposals go through proposed → approved → executed → rolled_back. Every write is attributed to an actor (human, or agent+version) resolved from its API token.

Workspace

Crate	What
incidental-core	Pure domain: event model, lifecycle FSM, signatures, confidence rules, projections, wire types. No IO.
incidental-server	incidental-api binary: axum HTTP API over Postgres (sqlx, embedded migrations).
incidental-client	Typed HTTP client over the wire types (use this from the agent).
incidental-cli	incidentctl binary: thin shell over the client.

CLI/API parity is by construction: every subcommand is one API call and --output json emits the API's wire types verbatim.

Quickstart (dev)

$ nix develop            # rust, postgres, jq, just, helm, treefmt
$ just db-start          # local postgres on 127.0.0.1:5433 (state in .dev/)
$ INCIDENTAL_BOOTSTRAP_TOKEN=dev-bootstrap just run-api &
$ export INCIDENTAL_TOKEN=dev-bootstrap
$ INCIDENTAL_TOKEN=dev-bootstrap just smoke   # end-to-end exercise

Typical agent session:

# mint a token bound to the agent identity (writes are attributed to it)
$ incidentctl token create --name kaos-prod --actor-kind agent \
    --actor-name kaos --actor-version 0.1.0

# a tick begins
$ RUN=$(incidentctl -o json run start --trigger periodic | jq -r .id)

# observe a condition; same signature never duplicates
$ incidentctl observe --resource-kind Pod --namespace media \
    --error-class CrashLoopBackOff --alert-name KubePodCrashLooping \
    --severity sev3 --title "jellyfin crashlooping" \
    --source kubectl --idempotency-key tick42-obs1

# investigate: hypotheses are a revision history, confidence is earned
$ incidentctl hypothesis "$INC" --kind proximate \
    --statement "bad ConfigMap rollout" --confidence probable \
    --ruled-out "memory pressure: RSS flat under load"

$ incidentctl evidence "$INC" --kind log-excerpt --source kubectl \
    --body-file - < excerpt.log

# act (after human approval per policy), resolve, close the loop
$ incidentctl action "$INC" --command "kubectl rollout undo deploy/jellyfin" \
    --effect "pods healthy" --run "$RUN"
$ incidentctl transition "$INC" --to resolved --cause fixed
$ incidentctl proposal submit --incident "$INC" --surface alerting \
    --title "alert on config-rollout crashloops" --body-file proposal.md

$ incidentctl run finish "$RUN" --report "1 incident resolved" --touched "$INC"

# later: how did we see it mid-incident? has it recurred?
$ incidentctl show "$INC" --as-of 2026-07-07T12:00:00Z
$ incidentctl signature "$FP"

API ≡ CLI surface

incidentctl	HTTP
observe	POST /v1/observations
list	GET /v1/incidents
show [--as-of]	GET /v1/incidents/{id}[?as_of=]
events	GET /v1/incidents/{id}/events
transition	POST /v1/incidents/{id}/transition
severity	POST /v1/incidents/{id}/severity
hypothesis	POST /v1/incidents/{id}/hypotheses
evidence	POST /v1/incidents/{id}/evidence
link	POST /v1/incidents/{id}/links
action	POST /v1/incidents/{id}/actions
comment	POST /v1/incidents/{id}/comments
signature	GET /v1/signatures/{fingerprint}
proposal submit/advance/list/show	POST /v1/proposals, POST /v1/proposals/{id}/advance, GET …
run start/finish/show/list	POST /v1/runs, POST /v1/runs/{id}/finish, GET …
token create/list/revoke	POST/GET /v1/tokens, DELETE /v1/tokens/{id}
health	GET /healthz (also /readyz, /metrics)

Wire contract: the serde types in incidental_core::wire (version v1). Writes accept expected_seq for optimistic concurrency (HTTP 409 on conflict) and idempotency keys where re-delivery is expected. CLI exit codes: 0 ok, 1 error, 2 usage, 3 not found, 4 conflict.

Lifecycle (incidentctl transition --to …):

detected → triaged → investigating → mitigating → resolved → closed
         ↘ suppressed        reopened ↗ (investigating)
         ↘ false-positive

Resolving requires --cause fixed|self-healed; the store keeps self-healed-then-recurring visible because it is intermittent-fault territory.

Auth

Static bearer tokens, SHA-256-hashed at rest, each bound to an actor identity. Bootstrap: start the API with INCIDENTAL_BOOTSTRAP_TOKEN=…, mint real tokens via incidentctl token create, then remove the variable.

Nix

$ nix build .#incidentctl      # CLI (also the default package)
$ nix build .#incidental-api   # API server
$ nix build .#oci-image        # docker-archive tarball running the API
$ nix run . -- --help          # run the CLI
$ nix flake check              # builds all crates + lefthook/treefmt checks

Push the image:

$ nix build .#oci-image -o result-oci
$ skopeo copy docker-archive:result-oci docker://registry.example.com/incidental-api:0.1.0

Helm

$ helm install incidental charts/incidental \
    --set image.registry=registry.example.com \
    --set database.existingSecret=incidental-db-app \
    --set bootstrap.existingSecret=incidental-bootstrap

The chart consumes a Postgres URI from an existing secret (CloudNativePG <cluster>-app convention, key uri), or templates its own CNPG Cluster with --set database.cnpg.enabled=true. Optional metrics.serviceMonitor.enabled=true wires /metrics into the Prometheus operator. Migrations run at API startup.

Spec §9 caveat: run the store (and its database) off the cluster it observes — it has to be up precisely when that cluster is on fire. The chart exists for deploying to a different cluster, or accepting the risk consciously.

Deferred (v2)

Suppression windows, per-agent-version confidence calibration, evidence retention tiers, and redaction on ingest (spec §§3–5) are deliberately not in v1; the schema leaves room for each.