Incidents and fixes

There are two ways an incident starts:

• The incident watcher checks linked resources every minute with read-only calls. When a rule fires, it opens an incident and Medic starts at once. An owner can turn the watcher off for the workspace in Settings, Approvals and limits.
• Report incident on the Incidents page. Pick the resources that look broken, say what you see if you like, and choose a severity or let Medic decide. Medic looks back 60 minutes from the report by default.

There is one open incident per group of related resources. A new signal or report that touches an open incident joins it instead of opening another, and an incident on a related resource can take over an older one (the older one shows as merged, with a link).

The watcher reads only what discovery found, through the same read-only access. A check that fails is skipped and logged; it never opens an incident by itself.

An investigation is a run you can follow live on the incident page. Medic works in this order:

1. Gather

   Logs, metrics, and state for the affected resources and their neighbours on the map: the database a service reads, the secret it uses, the log group it writes to, recent commits and workflow runs, and recent deploys. Each line is redacted and gets a stable evidence id.

2. Remember

   Looks for similar past incidents in this workspace. A close match's diagnosis or fix can be reused, and the incident says so.

3. Research

   Searches the web for the most distinctive error, keeping at most 3 results, which are listed on the incident.

4. Diagnose

   Medic, on NVIDIA Nemotron Ultra, writes the root cause from the evidence. Every claim must cite evidence ids. Claims that cite nothing real are dropped, and the incident lists what was dropped.

5. Propose

   One or more fixes, in order, each with its own approval. Then apply, verify, and remember.

The incident page shows:

• A one-line summary and the root cause, with a confidence of high, medium, or low.
• Each claim with links to the exact evidence lines it rests on.
• The triggering change when Medic found one: a commit, a deploy, a secret that changed, or a config change.
• A timeline with the signals and the changes around them.

Medic prefers a lasting fix in your repository, and adds a quick fix only when production is hurting now.

Pull requests

A fix pull request changes code or infrastructure code in your repository. The approval card shows the files and the diff. Approve and open PR opens it on a new groundcrew/fix-* branch. Groundcrew never pushes to your default branch and never merges. Your normal review, checks, and pipeline do the rest.

Emergency actions

When production is down, or degraded with a high-confidence cause and a known way back, Medic also proposes an emergency action and puts it first. The card shows the plan as before and after lines. Approve and run runs it once.

When one fix depends on another, the card says Runs after the first one. A failed emergency fix can be planned again with a new approval.

These need emergency fixes turned on for the connection, and by default only owners approve them (an owner can let approvers approve them too).

Each action reads the current state into a plan first. When it runs, it checks the state again: if it changed since you approved, nothing is written and the fix shows as failed with the reason. A changed Kubernetes ConfigMap or Secret gets a note instead of an action, because Groundcrew cannot restore what it never reads.

After a fix is applied, Medic re-checks the signals that opened the incident: the rollout, the health check, the error rate, and metrics such as database CPU and latency. When they are healthy the incident resolves with the time to detect and to recover. If they stay bad until the check times out, the incident fails and says what is still failing.

When Medic finds no fix, it waits 90 seconds for related signals before giving up, so a second symptom can join the incident and lead to a better diagnosis. You can stop an investigation at any time with Stop: its pending approvals are rejected and the incident closes as failed.

A rollback can bring production back while the cause is still in your repository. Groundcrew tracks that so the next deploy does not bring the failure back:

• A fix pull request that is still open when the incident resolves becomes a follow-up. Groundcrew watches it until it is merged and a successful deploy workflow on the base branch contains the merge.
• If the incident was resolved by a rollback with no fix pull request, and Medic knows the commit that caused it, the incident shows "Running on a rollback" until a deploy no longer contains that change. A person can dismiss it.

Follow-ups are checked read-only through the GitHub App, for 7 days. The Overview page counts incidents with a permanent fix still pending.

Filter the Incidents list by status, severity, and time range. To hear about incidents in Slack or by email, see Notifications.