mrbaeksang/deepcloak

---

The problem

You ask a research tool a question. Half the best sources sit behind a Bot Wall — Cloudflare, Datadome, a Turnstile, a reCAPTCHA. Every other tool gets a 403, silently drops those pages, and hands you a thinner report. You never even learn what it missed.

What DeepCloak does

When a plain fetch hits a Bot Wall, DeepCloak Escalates that one URL to a Stealth Fetch and Bypasses the wall — recovering the content other agents abandon. Then it tells you, at the bottom of every report, exactly how many walls it broke through.

It’s a thin, local-first orchestrator over two great projects: local-deep-research (the research loop) and CloakBrowser (the stealth browser). Use it as a CLI, an MCP server, or a Claude skill. MIT.

🌑 Why we built this

The open web is quietly closing. More of the best writing now sits behind a bot check, and AI research agents — the tools we increasingly trust to read the web for us — go blind at exactly those doors, without ever saying so. A report that silently skips every walled source isn’t neutral; it’s wrong in a way you can’t see.

DeepCloak’s stance is simple: your agent should be able to read what a person with a browser can read — and it should be honest about how it got there. It attempts a Stealth Fetch when needed and prints an Evidence Record of the result. The process runs locally; search engines receive queries, source websites receive requests, and a configured cloud LLM receives research context. Choose a local LLM when that context must stay on the machine. Capability and transparency, MIT-licensed, no lock-in.

✨ Why it’s different

Verified live — not mocked. The clip above is an unedited screen recording (captured with ffmpeg, no compositing) of a real deepcloak run against a local LLM (Qwen) + SearXNG — no API key. It Escalates on each Bot Wall and Bypasses 8 Cloudflare/Turnstile walls in one pass, then writes a cited report. Full clip: docs/media/demo-real.mp4; a raw asciinema session is also kept at docs/media/demo.cast. Wall counts vary per run (8–20) because the open web does.

🚀 Quickstart

pip install deepcloak
deepcloak setup                       # one-time: downloads the stealth browser
export OPENAI_API_KEY=...             # or ANTHROPIC_API_KEY / GEMINI_API_KEY — or --provider ollama
deepcloak "How does Cloudflare Turnstile detect bots?" --depth detailed --out report.md

You get a cited report.md ending with a 🛡️ Bypassed N bot-walled sources section, plus a report.md.evidence.json sidecar.

🧠 How it works

search (DuckDuckGo, no setup) ─▶ candidate URLs
        │
        ▼  for each page:
   plain fetch ─▶ Bot Wall detected? ──no──▶ use it (fast)
                        │ yes
                        ▼
                  Escalate ─▶ Stealth Fetch (CloakBrowser) ─▶ Bypass
        │
        ▼
research loop (local-deep-research) ─▶ cited report + Evidence Records

Stealth is heavy, so DeepCloak tries a cheap plain fetch first and only launches the stealth browser when it actually detects a Bot Wall (--stealth auto, the default). Use --depth detailed/report to fetch full pages where Bypasses happen.

🤖 Connect it to your agent (MCP)

DeepCloak runs as a stdio MCP server exposing deep_research(query, depth, provider, model), quick_summary(query, provider, model), get_evidence(run_id), and list_models(provider).

Claude Code — add to your project’s .mcp.json (an example ships in this repo):

{ "mcpServers": { "deepcloak": { "command": "deepcloak", "args": ["mcp"] } } }

Codex — add to ~/.codex/config.toml:

[mcp_servers.deepcloak]
command = "deepcloak"
args = ["mcp"]

Then your agent can call deep_research and read bot-walled sources directly. Prefer a slash-style skill? Drop skill/SKILL.md into ~/.claude/skills/deepcloak/.

⚙️ Configuration

Picking a model: list what a provider offers with deepcloak models --provider openrouter (works for every provider; local servers via --base-url), then pass --model . With several credentials in your environment and a real terminal, DeepCloak simply asks. Settings persist in ~/.config/deepcloak/config.toml (override with $DEEPCLOAK_CONFIG); precedence: CLI flags > environment > config file.

Run history: finished runs are saved to ~/.local/share/deepcloak/runs/ — browse with deepcloak runs, replay a report with deepcloak show (--json for the full record).

⚠️ Responsible use

DeepCloak Bypasses bot-detection. You are responsible for having the right to access whatever you fetch. robots.txt is ignored by default; pass --respect-robots to honor it (ADR-0002). Don’t use it to violate sites’ terms or the law.

🗺️ Roadmap

• More Bot Wall signatures + smarter Escalation heuristics
• More search backends beyond DuckDuckGo / SearXNG
• Cache Bypassed pages across runs
• Richer Evidence Record export (HTML / JSON schema)

Ideas welcome — start a Discussion or open a feature request.

🛠️ Built on

local-deep-research (MIT) + CloakBrowser (MIT), via pip — no vendored code. Domain glossary in CONTEXT.md; design decisions in docs/adr/; contributing guide in CONTRIBUTING.md.

📄 License

MIT — see LICENSE and NOTICE.