fareedkhan-dev/production-grade-mcp-agentic-system

🎯 What This Is

Most MCP tutorials end with a @tool decorator that returns "hello world". That is fine for a demo. It is not what ships.

This repository is a reference implementation of an MCP server designed to run in production: multi-tenant, authenticated, observable, rate-limited, cached, circuit-broken, and governed. It exposes a company’s heterogeneous data layer (Postgres, Elasticsearch, S3, vector DB) to AI agents as a single, secure tool surface, and ships with a four-agent support copilot (Planner → Retriever → Synthesizer → Critic) that uses it end to end.

The codebase is deliberately organised around twelve components that keep showing up on the 3 AM pager when teams skip them. Each one lives in its own module and can be read, replaced, or extended independently.

---

🏗️ Architecture Overview

---

🧩 The 12 Components

---

📖 Diving Deeper, Section by Section

Each diagram below links back to the corresponding section in the blog, where every line of code is walked through in detail.

---

🚀 Quick Start

Prerequisites

• Docker & Docker Compose
• Python 3.11+ (only for running the CLI locally)
• An Anthropic API key (for the agent layer)

1. Clone and Configure

git clone https://github.com/FareedKhan-dev/production-grade-mcp-agentic-system.git
cd production-grade-mcp-agentic-system
cp .env.example .env

Edit .env and set at minimum:

• ANTHROPIC_API_KEY — for the agent layer
• ATLAS_AUTH_JWKS_URL — your OAuth 2.1 provider’s JWKS endpoint (or leave default for dev)

2. Bring Up the Stack

docker compose up -d

That brings up the full local environment:

3. Run the Support Copilot CLI

pip install -e .

export ATLAS_MCP_URL=http://localhost:8080
export ATLAS_MCP_TOKEN=dev-token
export ATLAS_TENANT=acme
export ANTHROPIC_API_KEY=sk-ant-...

atlas-copilot "Why was the refund on order o_9002 for CUST-1001 delayed?"

You will see the four agents run end-to-end, the final draft printed with [S1][S2] citations, and a full trace summary including token counts, tool calls, and the run_id that ties back to Jaeger.

4. Connect from Claude Desktop / Cursor

Add this to your MCP host config:

{
  "mcpServers": {
    "production-mcp": {
      "type": "http",
      "url": "http://localhost:8080/mcp",
      "headers": {
        "Authorization": "Bearer ${ATLAS_MCP_TOKEN}",
        "X-Tenant-Id": "acme"
      }
    }
  }
}

---

📂 Repository Structure

.
├── 📄 README.md
├── 🐳 docker-compose.yml          # Full local stack: app + data + observability
├── 🐳 Dockerfile                  # Two-stage build, non-root runtime
├── 📜 LICENSE
├── 📦 pyproject.toml              # Dependencies, dev tools, CLI entry points
├── ⚙️  .env.example                # Every setting documented by component
│
├── 🔧 config/                     # Runtime configuration (hot-reloadable)
│   ├── http_allowlist.yaml       # Per-tenant outbound HTTP allowlist
│   └── policy.yaml               # YAML-driven authorization policies
│
├── 🚢 deploy/                     # Deployment sidecar configs
│   ├── otel/config.yaml          # OpenTelemetry Collector pipeline
│   ├── prometheus/prometheus.yml # Prometheus scrape targets
│   └── sql/init.sql              # Schema + RLS policies + seed data
│
├── 📚 docs/                       # Deep-dive documentation
│   ├── AGENT_SYSTEM.md           # Multi-agent orchestrator internals
│   ├── ARCHITECTURE.md           # The 12 components in detail
│   └── DEPLOYMENT.md             # K8s, Cloudflare Workers, bare-metal
│
├── 🧠 src/atlas_mcp/              # Main application source
│   ├── config.py                 # Centralized typed settings
│   ├── server.py                 # ⚡ Component 1: Transport & dispatch
│   │
│   ├── 🤖 agents/                 # Four-agent support copilot
│   │   ├── planner.py            # Emits retrieval plan JSON
│   │   ├── retriever.py          # Bounded tool-calling loop
│   │   ├── synthesizer.py        # Drafts reply with citations
│   │   ├── critic.py             # Approves or sends one revise
│   │   ├── orchestrator.py       # Wires the four agents together
│   │   ├── mcp_client.py         # Thin JSON-RPC MCP client
│   │   ├── memory.py             # STM (Redis) + LTM (vector)
│   │   └── cli.py                # atlas-copilot CLI entry point
│   │
│   ├── 🔐 auth/                   # Components 2 + 3
│   │   ├── oauth.py              # JWT + JWKS validation
│   │   ├── middleware.py         # Bearer token extraction
│   │   └── policy.py             # YAML-driven policy engine
│   │
│   ├── 🛡️  governance/             # Component 12
│   │   ├── tenant.py             # Tenant pinning middleware
│   │   └── approval.py           # Human-in-the-loop gate
│   │
│   ├── 🔧 tools/                  # Components 4 + 6
│   │   ├── registry.py           # In-memory tool index + discovery
│   │   ├── base.py               # Tool abstract base + metadata
│   │   ├── atomic/               # Level 1: one backend each
│   │   ├── composed/             # Level 2: deterministic chains
│   │   └── workflow/             # Level 3: multi-step procedures
│   │
│   ├── 🔄 reliability/            # Component 7
│   │   ├── circuit_breaker.py    # 3-state machine per tool
│   │   ├── retry.py              # Exponential backoff + jitter
│   │   └── atba.py               # Adaptive Timeout Budget Allocation
│   │
│   ├── 🚦 ratelimit/              # Component 8
│   │   └── limiter.py            # Redis token bucket (Lua-atomic)
│   │
│   ├── ⚡ cache/                   # Component 9
│   │   └── manager.py            # L1 + L2 cache with stampede lock
│   │
│   ├── 🧱 errors/                 # Component 10
│   │   └── framework.py          # Structured Error Recovery (SERF)
│   │
│   ├── 🔭 observability/          # Component 11
│   │   ├── tracing.py            # OpenTelemetry spans
│   │   ├── metrics.py            # Prometheus instruments
│   │   └── audit.py              # Structured JSONL audit log
│   │
│   └── ✅ validation/             # Component 5
│       └── schemas.py            # Tool call envelope
│
└── 🧪 tests/                      # Narrow tests, load-bearing properties
    ├── test_circuit_breaker.py   # State machine transitions
    ├── test_errors.py            # SERF wire format + retry semantics
    └── test_policy.py            # Deny-beats-allow + default-deny

---

🎨 Tech Stack

---

🧪 Testing

The test suite is deliberately narrow, covering the three load-bearing safety properties:

pip install -e ".[dev]"
pytest -v

• test_circuit_breaker.py — state machine transitions, retryable vs deterministic error classification
• test_errors.py — SERF wire format, retry semantics, MCP-level error data
• test_policy.py — default-deny, deny-beats-allow, glob matching, PII condition blocking

---

🛣️ Production Deployment

For running this in an actual production environment (managed Postgres, real OAuth provider, SIEM integration, Kubernetes), see docs/DEPLOYMENT.md.

Key swaps between local dev and production:

---

📚 Documentation

• 📖 Blog Walkthrough — Building a Production-Grade MCP Server (recommended starting point)
• 🏗️ docs/ARCHITECTURE.md — The 12 components in depth
• 🤖 docs/AGENT_SYSTEM.md — Multi-agent orchestrator internals
• 🚢 docs/DEPLOYMENT.md — Production deployment options

---

📜 License

MIT. See LICENSE.

---