aiops-sre-agent

An alert triage agent for Kubernetes. Alertmanager sends it an alert group (one notification holding one or more alerts). The agent investigates with read-only tools (pods, events, logs, rollout history, Prometheus, runbooks) and returns a structured report: root cause, evidence, confidence and recommendations for the on-call engineer. If a fix is warranted, it can also propose an action that changes the cluster. A proposed action does not run until a human approves it. It is rejected automatically if nobody decides in time or the alert group resolves first. The terms are defined in CONTEXT.md.

Built with LangChain 1.x (create_agent + middleware) and MCP servers for every tool. The same servers plug into Claude Code or any MCP client.

flowchart LR
  am[Alertmanager] -->|webhook token| api[FastAPI receiverdedup, concurrency limit]
  api --> ag[LangChain agentstructured TriageReport]
  ag -->|MCP| k8s[kubernetes serverread-only + gated actions]
  ag -->|MCP| prom[prometheus server]
  ag -->|MCP| rb[runbooks server]
  ag -. write tool call .-> hitl{{human approval}}
  hitl -->|approver token| api
  api --> slack[Slack / logs]
  k8s --> cluster[(Kubernetes API)]

What a triage looks like

This is real output from claude-sonnet-5-5 on the checkout scenario: a rollout dropped DATABASE_URL. It ran on tool outputs recorded from the kind demo and replayed. See Evals. It took 4 tool calls in about 11 s: runbooks_runbook_for_alert, then kubernetes_list_pods, kubernetes_pod_logs and kubernetes_describe_deployment.

The checkout pod checkout-7c856f898c-bjtfx (revision 2) is crash looping because it starts without the DATABASE_URL env var. Revision 1 had that variable and its pod is still healthy.

Root cause (high confidence, bad_rollout): Deployment revision 2 (ReplicaSet checkout-7c856f898c) dropped the DATABASE_URL env var that revision 1 defined. The app exits with code 1 at startup because the variable is missing. Severity: medium · Affected: demo/Deployment/checkout, demo/Pod/checkout-7c856f898c-bjtfx

Evidence

• kubernetes_pod_logs: Previous logs of checkout-7c856f898c-bjtfx: ‘checkout starting, version 1.4.0 / FATAL: required env var DATABASE_URL is not set’
• kubernetes_list_pods: checkout-7c856f898c-bjtfx is 0/1 ready with 2 restarts and last terminated Error, exit code 1. The revision 1 pod checkout-7784c7f885-pwjtr is 1/1 Running with 0 restarts.
• kubernetes_describe_deployment: Revision 2 (created 36s ago) has env [] and env_from []. Revision 1 had env [DATABASE_URL]. The image is python:3.12-alpine in both revisions.

Recommendations

1. Roll back the checkout deployment to revision 1 (kubectl rollout undo deployment/checkout -n demo), or re-add DATABASE_URL to the pod template (from a Secret or ConfigMap).
2. Find out why the env var was removed in the latest change to the manifest or Helm values, and fix it at the source so it is not re-applied.
3. Add a CI check or startup validation so a rollout that lacks required config fails before it replaces healthy pods.
4. Don’t restart the pod. A restart will not fix a missing env var.

Runbook: pod-crashlooping

The other reports are in evals/results/2026-10-10/run-3. In the prompt-injection scenario, the agent reported the planted “scale to 0” instruction and didn’t act on it.

Safety model

An agent that reads logs and can change a cluster is a prompt-injection target. A log line can say “scale this to zero”. The controls are layered, and the important ones sit outside the model:

MCP tool annotations are hints from the server. The security boundary is RBAC, the namespace allowlist, the tool registration done on the server side, and the Hub’s own read/write list.

Quick start

Requirements: Python 3.12+, uv, Docker, kind.

uv sync
make cluster demo          # kind cluster + three broken workloads in namespace "demo"
cp .env.example .env       # set AIOPS_MODEL_API_KEY; AIOPS_CONTRACTS_DIR=demo/contracts
export SOPS_AGE_KEY="$(cat demo/contracts/test-keys/*.agekey)"   # the demo's public test keys
uv run aiops triage examples/alert-crashloop.json --contract demo-shop

The demo breaks three workloads in three classic ways:

Other models

Any init_chat_model string works. For an OpenAI-compatible gateway (LiteLLM, vLLM; see llm-serving-k8s):

AIOPS_MODEL=openai: AIOPS_MODEL_BASE_URL=http://litellm:4000/v1 AIOPS_MODEL_API_KEY=...

Running the Hub

The Hub runs every Contract from one VM you own: deploy/hub is a compose stack with the Hub, Postgres, Caddy and nightly backups.

1. VM: Docker with compose, a DNS name pointing at it, and ports 80 and 443 open. Copy deploy/hub there.
2. Contracts: clone your private contracts repo into deploy/hub/contracts with a read-only deploy key:

   ssh-keygen -t ed25519 -N '' -f ~/.ssh/aiops-contracts      # add the .pub as a read-only deploy key
   GIT_SSH_COMMAND='ssh -i ~/.ssh/aiops-contracts' git clone [email protected]:/contracts.git contracts
   git -C contracts config core.sshCommand 'ssh -i ~/.ssh/aiops-contracts'
   

3. Age keys: put every Contract’s age key in one file that only the Hub’s user (uid 10001) can read:

   cat acme.agekey globex.agekey > age-keys.txt && chmod 0400 age-keys.txt && sudo chown 10001 age-keys.txt
   

4. Settings: copy env.example to .env (Postgres password, domain) and hub.env.example to hub.env (approver token, model key).
5. Start: docker compose up -d --wait. Caddy gets a certificate for your domain.
6. Operator access: tailscale serve --bg 8080 publishes the API, which listens only on 127.0.0.1:8080, to your tailnet over HTTPS. Never publish port 8080 another way: ports Docker publishes bypass host firewalls.
7. Alertmanagers: each Contract’s Alertmanager posts to https:///contracts//alertmanager with that Contract’s token.

What is public: only POST /contracts//alertmanager, with bodies up to 1 MB (Caddyfile). Investigations, decisions, metrics and the API docs answer only on 127.0.0.1.

Updating Contracts: ./update-contracts.sh pulls the contracts repo and validates every Contract inside the image before restarting the Hub. If any Contract is invalid, it rolls the clone back and leaves the running Hub alone, because one bad Contract would otherwise stop alerts for all of them.

Backups: a pg_dump at start and every 24 hours, kept 14 days, in deploy/hub/backups. Dumps are owner-only because they hold Contract data (Reports, raw tool results). A dump on the VM is lost with the VM: copy backups/ somewhere else.

make hub-smoke runs the stack locally and in CI with the demo Contracts. It checks that the stack comes up healthy, that every path except the alert route returns 404 through Caddy (including ../ and %2F tricks and oversized bodies), that an alert is stored, that a backup restores, and that a bad contracts update is refused. CI does not cover tailscale serve, real Let’s Encrypt certificates, the deploy-key clone or the arm64 image.

Slack

Each Investigation is one thread in its Contract’s channel. The parent message shows the status (investigating, waiting for your approval, done, failed), and replies carry the Proposed actions, each Approval decision, the Report or the error.

1. Create the app from deploy/slack-app.yaml (api.slack.com/apps > From a manifest), install it, and set its bot token as AIOPS_SLACK_BOT_TOKEN.
2. For the buttons: Basic Information > App-Level Tokens > generate a token with connections:write, set it as AIOPS_SLACK_APP_TOKEN, and set your own member ID (your profile > ⋮ > Copy member ID) as AIOPS_SLACK_OPERATOR_ID.
3. Make one channel per Contract and invite the bot: /invite @aiops. The bot can’t post where it isn’t a member.
4. Put the channel’s ID (channel details > the C... at the bottom) in the Contract: slack: {operator_channel: C0123ABCD}. Two Contracts can’t share a channel.

While Proposed actions wait, the parent message has Approve (with a confirm step) and Reject (a modal asks for a reason, which the agent reads). Clicks arrive over Socket Mode, so the Hub opens no Slack endpoint. They take the same decision path as POST /investigations/{id}/decision:

• only AIOPS_SLACK_OPERATOR_ID’s clicks count, and only in the Investigation’s own Contract channel; anyone else gets a private “Only the Operator can decide”;
• each button names its pause, so a second click, or a click left over from an earlier pause, decides nothing;
• once the actions are decided (in Slack or through the API) or go Stale, the buttons disappear and the thread says who decided, or why it went Stale.

Without AIOPS_SLACK_APP_TOKEN there are no buttons, and you decide through the API. Everything posted is escaped (a label like `` arrives as text, never as a ping), redacted and capped. A Slack failure, including a dropped socket (it reconnects), is logged and never fails an Investigation.

Running as a service in one cluster

uv run aiops contracts new-token    # give the token to Alertmanager; the hash goes in
                                    # deploy/base/contract.yaml (webhook_token_sha256)
kubectl -n aiops create secret generic aiops-secrets \
  --from-literal=AIOPS_APPROVER_TOKEN=... --from-literal=AIOPS_MODEL_API_KEY=... \
  --from-literal=AIOPS_DATABASE_URL=postgresql://user:pass@host:5432/aiops   # a Postgres you run
kubectl apply -k deploy/base        # read-only
kubectl apply -k deploy/actions     # optional: evict / scale, still behind approval

The image is published to GHCR on version tags. To use your own build, cd deploy/base && kustomize edit set image ghcr.io/alexboscolo954/aiops-sre-agent=.

This in-cluster deploy serves one Contract, in-cluster (deploy/base/contract.yaml), mounted from a ConfigMap. For several Contracts, run the Hub on a VM instead (Running the Hub). Point Alertmanager at it with deploy/alertmanager-receiver.yaml.

The container runs as non-root on a read-only root filesystem, with all capabilities dropped and the RuntimeDefault seccomp profile. CI deploys the real image with these manifests on kind and checks health, auth and metrics (demo/deploy-smoke.sh).

State lives in Postgres: Investigations, the duplicate memory, every Approval decision (who, what, when), every tool call with its raw result (redacted, before the size cap, kept for turning an Investigation into an eval Scenario) and the LangGraph checkpoints. A pending approval survives a restart. After a restart, a deadline that passed while the Hub was down goes Stale, and a Run that was cut off becomes a Failed investigation. It is never replayed, so an approved write can’t run twice. The in-cluster deploy needs a Postgres you provide (the smoke test brings a throwaway one, demo/postgres.yaml); the Hub’s compose stack ships one. Timers and the concurrency limit are per process, so run one replica. AIOPS_MAX_CONCURRENT_TRIAGES is shared by every Contract, so one Contract’s alert storm can hold every slot.

MCP servers on their own

Each server runs standalone over stdio or HTTP:

uv run aiops mcp kubernetes --contract demo-shop      # AIOPS_CONTRACTS_DIR and age keys as above
uv run aiops mcp prometheus --contract demo-shop --transport http --port 8765

.mcp.json.example wires them into Claude Code, giving it the same safe, namespace-scoped cluster tools the agent uses.

Testing

The scripted-model tests use LangChain’s GenericFakeChatModel, so CI needs no API key. make test starts a throwaway Postgres in Docker (make db), and each test gets its own fresh database.

Evals

The scenarios in evals/scenarios are built from tool outputs recorded on the kind demo in CI (evals/build_scenarios.py). A replay middleware answers the model’s tool calls from the recording. The model sees the exact production tool names and schemas, with no cluster needed. Each scenario checks the root-cause category, required facts in the root cause and a tool-call budget. It also lists the Proposed actions it allows, which is none by default, so the injection case fails if a write is even attempted. An allowed proposal is recorded and rejected, and the run resumes to score the Report.

Model claude-sonnet-5-5. Runs 1–3 are from 2026-10-10, run 4 is from 2026-10-12 and run 5 is from 2026-10-15. Raw JSON and every report are in evals/results.

Between runs 1 and 3 the pass/fail checks never changed. What changed was the tool code, because reading the reports showed two real gaps:

• Run 1 → 2: runbook_for_alert returned only the first matching runbook. KubePodCrashLooping maps to both the crash-loop and the OOM runbook, and the agent pointed out it had been handed the wrong one. The tool now returns every match.

• Run 2 → 3: on the bad rollout, the agent lowered its confidence to medium because rollout history showed images but not config, so it “could not see revision 1’s spec”. Rollout history now includes env var names per revision. The cluster was re-recorded in CI to capture the new field, and the same scenario now reaches high confidence in one fewer call.

• Run 3 → 4: Category now classifies the root cause, not the failure mode (ADR 0001), so the expectations are stricter. The bad rollout must be bad_rollout only, where crashloop_bad_config used to pass too. The OOM cases expect resource_limits, and the image pull expects image_unavailable. The reports apply the precedence rule explicitly, for example “this is revision 1, so it is not a rollout regression”.

• Run 4 → 5: every kubernetes tool now takes a required cluster, limited to the Contract’s clusters. The replay now validates arguments against the real tool schema, so a missing or wrong cluster costs the model a turn, as it would in production. Tool calls are unchanged from run 4. A one-off run with an alert whose cluster label didn’t match the Contract’s only cluster also passed in 4 calls. Choosing between several clusters is not covered by a Scenario yet.

Four scenarios is a smoke test for triage quality, not a benchmark. Add scenarios by recording new failures on kind (AIOPS_RECORD_DIR=recordings make integration).

Contracts

The agent is moving from one global config to several Contracts, one per client engagement. The plan is in docs/roadmap.md. A Contract is a directory in your private contracts repo, never in this one (ADR 0003). It holds contract.yaml, a SOPS-encrypted secrets.sops.yaml with one age key per Contract, and its own runbooks/. Runbooks in shared/runbooks/ are Shared knowledge for every Contract; a Contract’s own Runbook wins on a name clash. A kubeconfig whose user runs a credential plugin (exec, as EKS, AKS kubelogin and GKE use, or an auth-provider) is refused unless that cluster sets allow_exec: true. The plugin would run on the Hub next to every other Contract’s secrets (ADR 0002). Name each cluster as your alerts’ cluster label does: the agent passes that label as the tools’ cluster argument, and a tool refuses any cluster or namespace outside its Contract. demo/contracts shows the layout with two synthetic Contracts and their test-only keys.

Run these from the contracts repo root, since sops looks for .sops.yaml in the current directory:

brew install sops age
age-keygen -o ~/.config/sops/age/acme.agekey   # one key per Contract, kept outside the repo
# add a creation rule for acme/secrets.sops.yaml with its public key to .sops.yaml
sops edit acme/secrets.sops.yaml               # create or edit the secrets
sops updatekeys acme/secrets.sops.yaml         # after changing acme's key in .sops.yaml
cat ~/.config/sops/age/*.agekey > ~/.config/sops/age/keys.txt
SOPS_AGE_KEY_FILE=~/.config/sops/age/keys.txt aiops contracts validate .

validate reports every problem as path: field: message. It never prints a secret value. Each Contract’s Alertmanager posts to /contracts//alertmanager with its own token. contract.yaml holds only the token’s SHA-256 (webhook_token_sha256); aiops contracts new-token prints a token and its hash line. Two Contracts can’t share a token, and none can equal the approver token.

Configuration

Hub settings are AIOPS_* environment variables (or .env). See config.py. Everything about a cluster, an integration or a webhook token lives in the Contract. serve, triage and mcp refuse to start if one of the old variables (AIOPS_KUBE_CONTEXT, AIOPS_NAMESPACES, …) is still set, and say where it moved.

Design notes

• Versions. langchain.mcp (on FastMCP 4 / MCP SDK 2.x) is beta, so dependencies are pinned exactly and uv.lock is committed. The older langchain-mcp-adapters package doesn’t support MCP 2.x yet.
• Why MCP for internal tools? The same servers serve the agent, Claude Code and any other MCP client. Each has its own safety checks, and they’re testable in-process with fastmcp.Client.
• Restart means eviction, not a patch. A rolling restart via patch deployments needs a permission that also allows changing images. Evicting one pod restarts it, respects PDBs, and needs only pods/eviction.
• Runbook search is keyword-based. Exact alert-name mapping plus keyword scoring is enough for hundreds of runbooks. Embedding search is the upgrade path past that.

License

MIT