aiops-sre-agent

An alert triage agent for Kubernetes. Alertmanager sends it an alert group (one notification holding one or more alerts). The agent investigates with read-only tools (pods, events, logs, rollout history, recent changes, Prometheus, runbooks) and returns a structured report: root cause, evidence, confidence and recommendations for the on-call engineer. If a fix is warranted, it can also propose an action that changes the cluster. A proposed action does not run until a human approves it. It is rejected automatically if nobody decides in time or the alert group resolves first. The terms are defined in CONTEXT.md.

Built with LangChain 1.x (create_agent + middleware) and MCP servers for every tool. The same servers plug into Claude Code or any MCP client.

flowchart LR
  am[Alertmanager] -->|webhook token| api[FastAPI receiverdedup, concurrency limit]
  api --> ag[LangChain agentstructured TriageReport]
  ag -->|MCP| k8s[kubernetes serverread-only + gated actions]
  ag -->|MCP| prom[prometheus server]
  ag -->|MCP| rb[runbooks server]
  ag -. write tool call .-> hitl{{human approval}}
  hitl -->|approver token| api
  api --> slack[Slack / logs]
  k8s --> cluster[(Kubernetes API)]

What a triage looks like

This is real output from claude-sonnet-5-5 on the checkout scenario: a rollout dropped DATABASE_URL. It ran on tool outputs recorded from the kind demo and replayed. See Evals. It took 4 tool calls in about 11 s: runbooks_runbook_for_alert, then kubernetes_list_pods, kubernetes_pod_logs and kubernetes_describe_deployment.

The checkout pod checkout-7c856f898c-bjtfx (revision 2) is crash looping because it starts without the DATABASE_URL env var. Revision 1 had that variable and its pod is still healthy.

Root cause (high confidence, bad_rollout): Deployment revision 2 (ReplicaSet checkout-7c856f898c) dropped the DATABASE_URL env var that revision 1 defined. The app exits with code 1 at startup because the variable is missing. Severity: medium · Affected: demo/Deployment/checkout, demo/Pod/checkout-7c856f898c-bjtfx

Evidence

• kubernetes_pod_logs: Previous logs of checkout-7c856f898c-bjtfx: ‘checkout starting, version 1.4.0 / FATAL: required env var DATABASE_URL is not set’
• kubernetes_list_pods: checkout-7c856f898c-bjtfx is 0/1 ready with 2 restarts and last terminated Error, exit code 1. The revision 1 pod checkout-7784c7f885-pwjtr is 1/1 Running with 0 restarts.
• kubernetes_describe_deployment: Revision 2 (created 36s ago) has env [] and env_from []. Revision 1 had env [DATABASE_URL]. The image is python:3.12-alpine in both revisions.

Recommendations