wtznicy/my-websearch

---

One server, four ways to use it: connect an AI client over MCP, run a one-shot CLI query, keep a local daemon alive, or use the built-in diagnostics before going live.

Jump to: Quick start · MCP tools · CLI · Configuration · Diagnostics · Development

✨ Features

my-websearch is a full-stack search and technical documentation engine built specifically for AI coding agents (Claude Desktop, Cursor, Cherry Studio, Windsurf, Cline, ZCode, etc.). It requires zero paid API keys:

• 🌐 13-Engine Federated Search & Smart Routing
  Aggregates direct domestic engines (Bing, Baidu, CSDN, Juejin, Sogou), global engines (DuckDuckGo, Brave, Startpage, Exa), and specialized developer platforms (GitHub, StackOverflow, Hacker News, arXiv). Features automatic language-aware routing (auto), parallel multi-query execution (queries: string[]), URL deduplication, and automatic minResults cascading fallback.
• ⚡ Native Anti-Bot & Proof-of-Work Solvers (Zero-Browser Fast Path)
  Built on wreq-js with native Chrome TLS/HTTP2 handshake emulation and persistent session cookies. Includes pure-JS/Rust solvers for Startpage’s Anubis SHA-256 PoW challenge and DuckDuckGo’s d.js (isJsaChallenge) HTML5/arithmetic challenge, bypassing anti-bot walls in milliseconds without launching a 400MB browser. Real URL decryption for Sogou redirect links, Baidu Location, and Bing Base64 links.
• 📄 High-Purity Markdown Article Extraction & Dedicated Site Purifiers
  Combines Mozilla Readability with intelligent chrome noise stripping. Features dedicated purification rules for WeChat Official Accounts (mp.weixin.qq.com), StackOverflow questions & accepted answers, and transparent internal routing for CSDN & Juejin. Removes , , ``, and breadcrumbs while faithfully preserving article titles, code blocks, and GFM tables. Supports seamless pagination (startIndex).
• 📚 Unified Context7 Library Docs Lookup (queryDocs)
  Directly query official documentation with natural package names (e.g. "next.js", "vue") or Context7 IDs without needing a separate resolve step. Includes automatic candidate ranking, blind-guess 404 self-healing, and related alternatives discovery.
• 🐙 Extended GitHub Resource Fetcher
  fetchGithubReadme supports GitHub/Gitee repository READMEs, raw code blob files (/blob/...), and complete issue discussions with comments (/issues/...).
• 🌏 Zero-Config Network Detection & Clash Fake-IP Ready
  Auto-detects OS system proxies (Windows Registry / macOS scutil / Linux env). Automatically routes overseas engines through proxy while keeping domestic engines on fast direct lines. Pre-whitelists 198.18.0.0/15 for Clash TUN / Fake-IP compatibility while enforcing strict SSRF security bounds.
• 🔌 Versatile Deployment Modes
  Works seamlessly across MCP STDIO (Claude Desktop/Cursor), Streamable HTTP / SSE (Cherry Studio/remote clients), CLI one-shot commands, and a Local HTTP Daemon with connection pool reuse and Prometheus /metrics monitoring.

---

🏗️ Architecture

flowchart TB
    subgraph Clients["🤖 Entrypoints"]
        MCP["MCP Server(STDIO / Streamable HTTP / SSE)"]
        CLI["CLI Commands(my-websearch search / fetch-*)"]
        Daemon["Local Daemon(127.0.0.1:3210 · /health · /metrics)"]
    end

    subgraph Core["🧠 Search & Fetch Orchestrator"]
        Router["Language-Aware Auto RouterZH → Baidu/Sogou | EN/Tech → Bing + DuckDuckGo"]
        Cascade["minResults Cascade & Circuit Breaker(Auto Fallback + 5min TTL LRU Cache)"]
        Ranker["Cross-Engine URL Deduplication & BM25 Relevance Ranking"]
    end

    subgraph Transport["🛡️ Anti-Bot & Security Transport"]
        Wreq["wreq-js Chrome TLS/H2 Fingerprint+ Automatic Session Cookie Jars"]
        Solvers["Millisecond Challenge SolversStartpage PoW | DDG JSA Solver"]
        PW["Playwright Stealth Browser Fallback(Budget Bounds + Cross-Process Locking)"]
        Guard["SSRF Guard & Clash Fake-IP Support(PROXY_ENGINES Split Routing + 198.18.0.0/15)"]
    end

    subgraph Engines["🌍 13 Search Engines + 6 Content/Docs Tools"]
        CN["🇨🇳 Direct Domestic EnginesBing · Baidu · CSDN · Juejin · Sogou"]
        INTL["🌐 Global & Tech EnginesDuckDuckGo · Brave · Startpage · ExaGitHub · StackOverflow · HackerNews · arXiv"]
        Docs["📚 Content & Official DocsfetchWebContent · queryDocs · GitHub/Gitee · CSDN/Juejin"]
    end

    Clients --> Core
    Core --> Transport
    Transport --> Engines

---

🌍 13 Search Engines

---

🛠️ 4 Core MCP Tools (Minimal & Token-Efficient)

💡 Backward Compatibility: Legacy fetchCsdnArticle and fetchJuejinArticle are automatically routed under fetchWebContent. Legacy resolveLibraryId is merged into queryDocs (which supports directly passing library name or legacy libraryId alias). Set REGISTER_LEGACY_ARTICLE_TOOLS=true if your LLM requires fetchCsdnArticle / fetchJuejinArticle explicitly exposed as separate tools.

---

🚀 Quick Start

1. Run with NPX (Zero Configuration)

No installation required:

npx -y my-websearch@latest

💡 Automated Proxy Discovery: The server automatically detects your operating system’s proxy settings. If a local proxy (Clash, v2ray, Surge) is active, overseas engines automatically use the proxy while domestic engines stay on fast direct lines. 198.18.0.0/15 is whitelisted out of the box for Clash Fake-IP.

2. Configure in AI Clients

🔹 Claude Desktop / Cursor / Windsurf / Cline (STDIO Mode)

Add to your MCP settings file:

{
  "mcpServers": {
    "my-websearch": {
      "command": "npx",
      "args": ["-y", "my-websearch@latest"],
      "env": {
        "MODE": "stdio"
      }
    }
  }
}

(For Windows environments where npx is not in the system path, set command to cmd and args to ["/c", "npx", "-y", "my-websearch@latest"])

🔹 Cherry Studio (Streamable HTTP Mode)

Start the server in your terminal with npx my-websearch@latest (default port 3211), then configure:

{
  "mcpServers": {
    "web-search": {
      "name": "MyWebSearch",
      "type": "streamableHttp",
      "baseUrl": "http://localhost:3211/mcp"
    }
  }
}

---

💻 CLI & Local Daemon

my-websearch can be used directly from your terminal as a standalone CLI tool or persistent daemon. Use --json in automation and keep the daemon on 127.0.0.1 unless you explicitly need remote access:

# Global install
npm install -g my-websearch

# Start persistent local HTTP daemon (port 3210 by default)
my-websearch serve

# One-shot terminal search (reuses running daemon automatically)
my-websearch search "Model Context Protocol best practices" --limit 5 --min-results 5 --json

# Extract web page as clean Markdown
my-websearch fetch-web "https://blog.vuejs.org/posts/vue-3-5" --max-chars 15000 --json

# View daemon status and Prometheus metrics (/health, /status, /metrics)
my-websearch status --json

# Clear in-memory search cache
my-websearch cache-clear

---

🩺 Diagnostics

Use the built-in doctor command before troubleshooting a proxy, browser, or engine:

# Local configuration and browser checks
my-websearch doctor

# Real proxy/engine probes (may contact upstream services)
my-websearch doctor --network

# Limit network probes to selected engines
my-websearch doctor --network --engine bing,duckduckgo --json

# Reset and clear cached engine circuit breaker / health state
my-websearch doctor --reset-health

The report separates configuration, proxy TCP reachability, HTTP reachability, application parsing, browser launch, and anti-bot challenges. Network mode performs real requests and should not be used as a high-frequency health check.

⚙️ Configuration & Environment Variables

Global Configuration (~/.my-websearch/config.json)

To share settings across all MCP clients without duplicate configuration:

{
  "apiKeys": {
    "context7": "ctx7sk-...",
    "exa": "..."
  },
  "proxy": {
    "url": "http://127.0.0.1:7897",
    "useProxy": false,
    "engines": ["duckduckgo", "exa", "brave", "startpage"]
  }
}

📋 Exhaustive Environment Variables Reference

Can be set via the global configuration file (~/.my-websearch/config.json), local .env, or system environment variables:

1. 🔍 Search & Routing Orchestration

2. 🌐 Proxy & Network Security

3. 🔐 Credentials & API Keys

4. 🖥️ Server Modes, Concurrency & Session Reaper

5. 🧭 Playwright Browser & Stealth Controls

[!NOTE] BING_PLAYWRIGHT_FALLBACK and STARTPAGE_PLAYWRIGHT_FALLBACK have been consolidated into SEARCH_MODE (e.g. SEARCH_MODE=bing or SEARCH_MODE=request). Backward compatibility is automatically preserved.

6. 🛠️ Application Directory & Tool Customization

💡 Zero-Config Upgrades (Redundant Variables Removed):

• CORS Switches (ENABLE_CORS/CORS_ORIGIN): CORS is disabled by default; enable it explicitly and restrict origins when browser clients need cross-origin access.
• Playwright Package Selection (PLAYWRIGHT_PACKAGE/PLAYWRIGHT_MODULE_PATH): Now 100% automated via intelligent client and system binary discovery.
• Remote Browser Endpoint (PLAYWRIGHT_WS_ENDPOINT): Unified into standard PLAYWRIGHT_CDP_ENDPOINT.
• Emoji Switch (OPEN_WEBSEARCH_NO_EMOJI): Automatic console code page detection smoothly falls back to ASCII tags on legacy terminals.
• Fragmented Debug Flag (OPEN_WEBSEARCH_DEBUG): Unified into standard LOG_LEVEL=debug.
• Cookie Persistence TTL (MYWEBSEARCH_COOKIE_TTL_DAYS): Solidified to 30 days with automated sliding renewal.

---

🤝 Contributing & Tests

Pull requests and issues are welcome! To run the test suite locally:

npm install
npm run build
npm test              # Run the Vitest suite (45 files, 401 tests at the current baseline)
npm run test:live     # Run bounded-concurrency live integration tests
npm run eval:live     # Run 23-case live STDIO JSON-RPC evaluation suite

Acknowledgements

Author: wtznicy

This project evolved from Open-WebSearch (originally created by Aas-ee) — special thanks to the original author. Thanks also to:

• wreq-js for native Chrome TLS/HTTP2 fingerprint impersonation
• Context7 for powering official documentation retrieval
• Mozilla Readability & Turndown for content extraction and GFM Markdown formatting