MCP Data Gateway

Status: OAuth Provider live. Phases 1–9.5 shipped — stdio, HTTP, OAuth 2.0 Authorization Server (RFC 8414/7591/7636/9728), per-user multi-tenant sessions, CORS, STDIO keyring auth, and LLM-facing endpoint metadata. Verified end-to-end against Claude Desktop (STDIO + natural language) and MCP Inspector (HTTP

• OAuth dance against a real Service API). 432 passing tests. Public-deploy recipe (Phase 10) is the next milestone. See docs/testing-user-ux.md for the test playbook, the Development Roadmap, or docs/plan.md.

A Python-based Model Context Protocol (MCP) server that acts as a unified data gateway, enabling Claude (and other MCP clients) to send and receive data across multiple external APIs through a single, secure interface.

Overview

This MCP server provides:

• Generic data handling for multiple data types
• Generic API gateway supporting any REST or GraphQL endpoint
• OAuth 2.0 authentication via an operator-run scripts/oauth_login.py (PKCE flow, 127.0.0.1 callback)
• Secure credential storage using system keyring
• Foundation for MCP App evolution in the future

Features

Architecture

Files marked (implemented ✓) exist today. Files marked (planned) are upcoming phases.

MCP/
├── src/
│   ├── server.py              # MCP server entry point (implemented ✓)
│   ├── auth/                  # OAuth 2.0 + keyring (implemented ✓)
│   │   ├── oauth.py           # PKCE auth-code flow, callback server bound to 127.0.0.1
│   │   └── credentials.py     # Keyring-backed store with peek/get/store/clear
│   ├── gateway/               # REST/GraphQL HTTP client (implemented ✓)
│   │   ├── api_client.py      # RestClient + GraphQLClient with retry, redacted logging
│   │   └── handlers.py        # Response normalization, size enforcement, error mapping
│   ├── models/                # Pydantic data models (planned)
│   │   └── data_models.py
│   ├── tools/                 # MCP tool definitions (implemented ✓)
│   │   ├── builtin.py         # list_apis tool
│   │   ├── registry.py        # ToolRegistry / ToolSpec
│   │   ├── context.py         # ToolContext dependency container
│   │   ├── auth_resolver.py   # auth.type branching (oauth2/bearer/api_key/None)
│   │   └── mcp_tools.py       # fetch_data/send_data/execute_graphql/get_status
│   ├── config.py              # API config loader with ${VAR} substitution (implemented ✓)
│   ├── events/                # Activity logging (implemented ✓)
│   │   ├── schemas.py         # Pydantic models (audit/debug/usage/insight)
│   │   ├── redaction.py       # Sensitive data redaction
│   │   ├── retention.py       # Per-month file rotation cleanup
│   │   ├── writers.py         # Async JSONL writer + queue
│   │   └── recorder.py        # Public Recorder API
│   └── transport/             # Transport layer (Phase 8 — implemented ✓)
│       ├── stdio.py           # stdio transport (default)
│       └── http.py            # Streamable HTTP + Bearer middleware + loopback guard
├── config/
│   ├── api_configs.json       # API service configurations (committed; uses ${VAR} placeholders)
│   └── api_configs.example.json  # Template with all four auth.type variants
├── docs/
│   └── plan.md                # Implementation plan / roadmap
├── scripts/                   # Operator CLI helpers (implemented ✓)
│   └── oauth_login.py         # Drive OAuth 2.0 + PKCE flow, persist token in keyring
├── tests/
│   ├── auth/                  # Unit tests for src/auth/ (49 cases — implemented ✓)
│   ├── events/                # Unit tests for src/events/ (27 cases, 51 collected w/ parametrize)
│   ├── gateway/               # Unit tests for src/gateway/ (61 cases — implemented ✓)
│   ├── tools/                 # Unit tests for src/tools/ (37 cases — implemented ✓)
│   ├── scripts/               # Unit tests for scripts/ (15 cases — implemented ✓)
│   ├── transport/             # Unit tests for src/transport/ (23 cases — implemented ✓)
│   ├── integration/           # Full-flow + subprocess smoke tests (5 cases — implemented ✓)
│   ├── test_config.py         # Config loader tests
│   ├── test_server.py         # Server bootstrap + _build_oauth_configs tests
│   └── test_example_config.py # Schema-drift guard for api_configs.example.json
├── .claude/commands/          # Slash commands for the dev workflow
│   ├── generate-prp.md        #   /generate-prp INITIAL.md  → PRPs/{feature}.md
│   └── execute-prp.md         #   /execute-prp PRPs/{...}   → implements + validates
├── PRPs/
│   ├── templates/prp_base.md  # Template each PRP fills in
│   └── {feature}.md           # Generated implementation blueprints
├── INITIAL.md                 # Per-feature scope delta (input to /generate-prp)
├── .env.example               # Environment variables template
├── .gitignore                 # Excludes secrets and build artifacts
├── pyproject.toml             # pytest + ruff + mypy configuration
├── requirements.txt           # Runtime dependencies
├── requirements-dev.txt       # Dev/test deps (pytest, ruff, mypy)
├── CLAUDE.md                  # Project rules + Context Engineering workflow
└── README.md                  # This file

MCP Tools

Per-module responsibilities and detailed module-by-module breakdown: docs/plan.md § Architecture Overview. Activity logging contract (four categories, retention, redaction): CLAUDE.md § Activity Logging.

Authentication Flow

OAuth login is operator-initiated (one-time, per provider) and tools are read-only with respect to credential acquisition — they never auto-open the browser. This split keeps the request path predictable for clients (Claude Desktop, Codex CLI, etc.) and concentrates the browser flow in a single CLI surface that’s easy to script and test.

A) First-time login (operator runs once per provider)
─────────────────────────────────────────────────────
$ python -m scripts.oauth_login github
        ↓
1. Script builds authorize URL with PKCE
2. Browser opens at provider's authorize page
3. User clicks "Authorize"
4. Callback at http://127.0.0.1:8765/callback receives auth code
5. Script exchanges code for tokens
6. TokenInfo persisted in OS keyring (service=mcp-data-gateway, account=)

B) Subsequent tool calls (auto, no UI)
──────────────────────────────────────
1. Claude invokes tool (e.g., fetch_data)
        ↓
2. MCP checks credentials in keyring
        ↓
3a. Valid token                                       →  proceed with API call
3b. Token < 5 min from expiry, refresh_token present  →  silent refresh, then proceed
3c. No token / expired with no refresh                →  return AUTH_REQUIRED *
        ↓
4. Tool returns response (data + metadata, or {error: AUTH_REQUIRED})

* On AUTH_REQUIRED, operator re-runs `python -m scripts.oauth_login `
  and then retries the original tool call.

Tech Stack

• Python 3.10+
• mcp — Model Context Protocol Python SDK
• httpx — Async HTTP client (REST + GraphQL)
• keyring — Cross-platform secure credential storage
• pydantic — Data validation and modeling
• python-dotenv — Environment variable management
• uvicorn + starlette — ASGI server + framework for the HTTP transport (Phase 8)

Quickstart

Prerequisites: Python 3.10+ and pip (or uv).

# 1. Clone and enter the repo
git clone https://github.com/Chawengwit/MCP.git mcp-data-gateway
cd mcp-data-gateway

# 2. Create a virtual environment
python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate

# 3. Install runtime + dev dependencies
pip install -r requirements.txt
pip install -r requirements-dev.txt # only if you want to run pytest

# 4. Copy the env + config templates and edit them
cp .env.example .env                       # add provider client_id / secrets
cp config/api_configs.example.json config/api_configs.json

# 5. Run the server (talks MCP over stdio)
python -m src.server

The server boots, loads config/api_configs.json, starts the Recorder, builds the ToolContext, and registers all five tools (list_apis, fetch_data, send_data, execute_graphql, get_status). On SIGINT/SIGTERM the Recorder queue drains and the server exits cleanly.

First-run note. With the default example config the server will warn “Skipping OAuth config for example_rest_api: missing client_id, client_secret” until you populate EXAMPLE_REST_CLIENT_ID and EXAMPLE_REST_CLIENT_SECRET in .env. The other tools (list_apis, get_status, plus any bearer / api_key / no-auth APIs) work without OAuth setup.

Configuring an API

The shipped config/api_configs.example.json covers all four auth.type paths:

Never commit literal secrets — every credential field in the file uses a ${ENV_VAR} placeholder. The config loader substitutes from your .env (or the process environment) at startup.

Environment variables

See .env.example for the full annotated template. Most-used variables:

Usage

Running Tests

# After installing requirements-dev.txt:
pytest tests/

# Run a specific test file with verbose output
pytest tests/events/test_writers.py -v

# Currently 288 tests passing across src/events/, src/auth/, src/gateway/, src/tools/, src/transport/, scripts/, plus integration + example-config drift tests.

Running the MCP Server

python -m src.server

The server boots, loads config/api_configs.json, starts the Recorder, builds the ToolContext, and registers all five tools (list_apis, fetch_data, send_data, execute_graphql, get_status). On SIGINT/SIGTERM the Recorder queue drains and the server exits cleanly.

Connecting to Claude Desktop

server.py performs its own bootstrap (sys.path + os.chdir + load_dotenv), so a cwd field in the client config is not required — point args at the script directly:

{
  "mcpServers": {
    "data-gateway": {
      "command": "/path/to/mcp-data-gateway/.venv/bin/python",
      "args": ["/path/to/mcp-data-gateway/src/server.py"]
    }
  }
}

On macOS, this lives at ~/Library/Application Support/Claude/claude_desktop_config.json. After editing, fully quit and relaunch Claude Desktop; the 🔌 menu should list data-gateway with all five tools.

Connecting to OpenAI Codex CLI

Codex CLI uses TOML at ~/.codex/config.toml:

[mcp_servers.data-gateway]
command = "/path/to/mcp-data-gateway/.venv/bin/python"
args = ["/path/to/mcp-data-gateway/src/server.py"]

Verified against codex CLI — list_apis, get_status, and fetch_data round-trip identically to Claude Desktop.

Other MCP clients (Cursor, Cline, etc.)

Any client that supports stdio MCP servers will work with the same shape: a command and an args list.

HTTP transport (ChatGPT Connectors, MCP Inspector, custom HTTP clients)

Set MCP_TRANSPORT=http to expose the server as a Streamable HTTP endpoint instead of stdio.

# Generate a strong bearer token
export MCP_HTTP_BEARER_TOKEN="$(python -c 'import secrets; print(secrets.token_hex(32))')"

# Run on loopback (default host/port)
MCP_TRANSPORT=http \
MCP_HTTP_HOST=127.0.0.1 MCP_HTTP_PORT=8080 \
python -m src.server

The server prints its bind address to stderr and serves the MCP protocol at POST /mcp, GET /mcp, and DELETE /mcp.

Test from curl:

# Without the token → 401
curl -i -X POST http://127.0.0.1:8080/mcp \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize"}'

# With the token → initializes a session
curl -i -X POST http://127.0.0.1:8080/mcp \
  -H "Authorization: Bearer $MCP_HTTP_BEARER_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize",
       "params":{"protocolVersion":"2025-11-25","capabilities":{},
                 "clientInfo":{"name":"curl","version":"1.0"}}}'

The response includes a Mcp-Session-Id header — echo it on every subsequent request via -H "mcp-session-id: ".

Connect from MCP Inspector (great for interactive debugging):

npx @modelcontextprotocol/inspector
# UI opens at http://localhost:6274
# Transport: Streamable HTTP
# URL:       http://127.0.0.1:8080/mcp
# Auth:      Bearer 

Connect from ChatGPT Custom Connectors: paste the public URL of the server (via reverse proxy / Cloudflare Tunnel / ngrok) and the bearer token. ChatGPT talks server-to-server, so no CORS configuration is needed.

Loopback safety guard. The server refuses to start if MCP_HTTP_HOST is non-loopback (0.0.0.0, public IP, etc.) and MCP_HTTP_BEARER_TOKEN is unset or empty — preventing accidental unauthenticated public binds. Set the token, bind to 127.0.0.1, or enable the OAuth Provider (see below).

OAuth Provider — Claude.ai “Add custom connector” (Phase 9)

When MCP_OAUTH_ENCRYPTION_KEY is set, the HTTP transport additionally exposes an OAuth 2.0 Authorization Server so Claude.ai (and any MCP client that speaks OAuth 2.1 + RFC 9728) can connect.

# 1. Generate a Fernet key once and add it to .env. Rotating it later invalidates
#    every stored Service API session — keep it stable.
python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"

# 2. Configure the Service API entry in config/api_configs.json with
#    auth.type=session_login. See config/api_configs.example.json for the shape.

# 3. Start the server with OAuth Provider on (loopback or ngrok-tunnelled HTTPS).
MCP_TRANSPORT=http \
MCP_OAUTH_ENCRYPTION_KEY= \
MCP_OAUTH_ISSUER=https://.ngrok-free.app \
python -m src.server

# 4. In Claude.ai, "Add custom connector" → paste the issuer URL.
#    Claude.ai will discover /.well-known/oauth-authorization-server,
#    POST /register, then redirect users to /authorize. Each user pastes
#    their Service API api_key + secret_key into the consent form;
#    the encrypted session is stored under their Service API user_id.

Operator inspection (no plaintext output):

python -m scripts.oauth_admin list-clients
python -m scripts.oauth_admin list-tokens
python -m scripts.oauth_admin list-sessions
python -m scripts.oauth_admin revoke-token --token 

The Phase 8 static-bearer path is unchanged when the OAuth Provider is on; both auth methods are accepted at /mcp. OAuth tokens carry a user_id into audit/usage/insight logs, static-bearer tokens carry user_id=null.

Example Interactions

Once connected, Claude can:

• List configured APIs: “Show me the available API services”
• Fetch data: “Get the user list from example_api”
• Send data: “Create a new record in example_api with this data…”
• Execute GraphQL: “Run this GraphQL query against my API…”

The first time Claude uses a tool requiring authentication, your browser will open automatically for OAuth login.

Response Format

All MCP tools return structured JSON: {data, metadata} on success, {error} on failure. Large responses truncate (success + cursor) where safe; binary/streaming emit RESPONSE_TOO_LARGE. GraphQL surfaces partial-success (data + errors).

For the full spec — exact field shapes, the error-code table, the truncation rule, and the GraphQL handling — see CLAUDE.md § Response Format Conventions.

OAuth Setup

For each oauth2 API in api_configs.json:

1. Register an OAuth application with the provider (GitHub OAuth Apps, Google Cloud OAuth client, etc.) and set the redirect URI to http://127.0.0.1:8765/callback exactly. Use 127.0.0.1, not localhost — browsers may treat them as different origins for OAuth state tracking.
2. Copy the resulting client ID and client secret into .env under names matching your api_configs.json placeholders. By convention: ${API_ID_UPPER}_CLIENT_ID and ${API_ID_UPPER}_CLIENT_SECRET — e.g. GITHUB_CLIENT_ID / GITHUB_CLIENT_SECRET for api_id="github".
3. Run the operator login helper once per provider:

   python -m scripts.oauth_login github            # initial login
   python -m scripts.oauth_login github --clear    # delete + re-auth
   

   The script opens your browser at the provider’s authorize URL, captures the auth code via the one-shot 127.0.0.1 callback server, exchanges it for tokens, and stores them in your OS keyring. Subsequent runs reuse the stored token; the gateway auto-refreshes when ≤ 5 minutes remain.

Why a separate script? MCP tools (fetch_data, send_data, execute_graphql) intentionally do not auto-trigger the browser flow — they return AUTH_REQUIRED so the client (Claude / Codex / etc.) can surface it cleanly. The login script is the operator-side counterpart.

If the default port 8765 is in use on your machine, set OAUTH_CALLBACK_PORT= in .env and update the redirect URI you registered with the provider.

Keyring Setup (per OS)

Tokens are stored in the OS-native secure keyring. No additional setup is needed on most desktops:

If keyring is unavailable, Credentials raises CredentialStorageError("No keyring backend available. Install 'keyrings.alt' …") on the first OAuth API call. The error is fail-loud by design — the gateway will not silently fall back to a less secure store.

Troubleshooting

For the full failure-modes table that this section is derived from, see CLAUDE.md § Common Failure Modes.

Logging (Operator-Only)

The Recorder writes one append-only JSONL file per (category, month) under $MCP_LOG_DIR (default ./logs/):

logs/
├── audit/   YYYY-MM.jsonl   who/when/what — security & compliance
├── debug/   YYYY-MM.jsonl   full HTTP exchange (redacted) — troubleshooting
├── usage/   YYYY-MM.jsonl   per-call latency / sizes — analytics
└── insight/ YYYY-MM.jsonl   tool args + response summaries — Claude's request patterns

• Rotation is automatic per month; the current month is never deleted.
• Old files are pruned on server startup according to MCP_LOG_RETENTION_DAYS (default 365).
• All four streams pass through the central redaction helpers (src/events/redaction.py); Authorization headers, access_token, refresh_token, client_secret, password, api_key, secret, and any per-API redact_fields are replaced with ``.

These logs are operator-only. No MCP tool exposes them to Claude. Treat the directory like any production audit log: review before sharing, retain per your org’s policy, and back up if you need historical analytics.

To enable verbose request tracing temporarily:

MCP_LOG_DEBUG_ENABLED=true MCP_LOG_LEVEL=DEBUG python -m src.server

All logs go to stderr (stdout is reserved for the MCP JSON-RPC protocol).

For the full debug/logging strategy and env-var reference, see CLAUDE.md § Debug & Logging Strategy.

Development Workflow

This project uses a Context Engineering workflow for non-trivial features. The full description lives in CLAUDE.md § Context Engineering Workflow. Quick summary:

1. Edit INITIAL.md          ← describe ONE feature (delta vs docs/plan.md)
2. /generate-prp INITIAL.md ← AI researches and writes PRPs/{feature}.md
3. /execute-prp PRPs/{...}  ← AI implements + runs ruff/mypy/pytest until green

src/events/ is the project’s reference implementation — new code mirrors its patterns. See CLAUDE.md § Reference Implementation.

For small fixes (single-line changes, doc edits, etc.) skip the workflow and edit directly.

Development Roadmap

Per-phase deliverables and verification plan: docs/plan.md. Operator playbook for testing both transports end-to-end: docs/testing-user-ux.md. Future scalability ideas (multi-tenant, caching, additional transports) live in docs/plan.md § Future Scalability.

Security

• All credentials stored in OS-level secure keyring (Keychain on macOS, Credential Manager on Windows, Secret Service on Linux)
• .env file excluded from version control via .gitignore
• OAuth uses standard authorization code flow with PKCE (no implicit grant)
• Tokens never logged or exposed in error messages — Pydantic Field(repr=False) keeps secret fields out of repr() and f-string output
• OAuthConfig rejects non-HTTPS authorize_url / token_url at validation time
• Local callback server binds to 127.0.0.1 (not localhost) and only during the OAuth flow; closes immediately after the auth code is received

License

TBD

Contributing

Phases 1–9.5 are shipped — STDIO + HTTP transports, GitHub OAuth, the full OAuth Provider (RFC 8414/7591/7636/9728), keyring-backed STDIO session_login, CORS, and LLM-facing endpoint metadata are all in main with 432 passing tests. The active feature delta is Phase 10 — Production Deploy Recipe (Dockerfile + Caddy + VPS guide) in INITIAL.md; implementation goes through /generate-prp INITIAL.md → /execute-prp PRPs/{...}.md (see the Development Workflow section). For bug fixes and small edits, open a PR directly. Contribution guidelines (style, commit format, review process) will be formalized as the project gains contributors.