OmniSearch

A comprehensive Model Context Protocol (MCP) server built with FastMCP that provides LLMs with real-time, high-fidelity access to the web. This server aggregates multiple search engines, social platforms, and developer tools into a single interface, allowing AI agents to perform deep research, track community sentiment, and analyze technical documentation.

Design docs → wiki — tool selection guide, decision matrix, recommended workflows, tools status & known quirks, plugin setup, and development standards.

---

🚀 Features

The server provides a diverse suite of tools categorized by their primary use case:

🌐 General Web Search & Retrieval

💬 Social & Community Intelligence

🎓 Academic & Reference

---

📋 Prerequisites

Optional External Tools

---

⚙️ Installation

You have three options depending on your use case:

Option A: Quick Run (via uvx)

Fastest way to try it out without cloning the repo. Add to your MCP client config:

{
  "mcpServers": {
    "OmniSearch": {
      "command": "uvx",
      "args": [
        "--from",
        "git+https://github.com/sydasif/web-search-mcp.git",
        "omnisearch"
      ]
    }
  }
}

Option B: Permanent Install

Fastest startup times with a globally installed tool:

uv tool install git+https://github.com/sydasif/web-search-mcp.git

Then configure your MCP client:

{
  "mcpServers": {
    "OmniSearch": {
      "command": "omnisearch"
    }
  }
}

Option C: Development Install

If you want to modify the code or contribute:

git clone https://github.com/sydasif/web-search-mcp.git
cd web-search-mcp
uv sync
uv run omnisearch

Verify It’s Working

Once the server is running, try a simple search:

search_web(query="current weather in Tokyo")

---

🔐 Configuration & Authentication

Most tools work out of the box with zero configuration. The following environment variables are only needed for premium or authenticated features.

Environment Variables Reference

Setting Up GitHub Authentication

Option 1 — Recommended: Use gh CLI

gh auth login

The server detects your local session automatically.

Option 2: Manual Token

export GITHUB_TOKEN="ghp_your_token_here"

Setting Up X/Twitter Authentication

X/Twitter search requires either session cookies or an API key.

Option 1 — Session Cookies (Bird CLI):

1. Log into x.com in your browser.
2. Open DevTools (F12) → Application (or Storage) → Cookies → x.com.
3. Copy the values for auth_token and ct0.
4. Export them in the shell where the MCP server runs:

   export AUTH_TOKEN="your_auth_token"
   export CT0="your_ct0"
   

   Note: These are session cookies. If searches return 401s, refresh them by logging out and back in.

Option 2 — Xquik API Key (Recommended):

1. Sign up at xquik.ai to get an API key.
2. Export it:

   export XQUIK_API_KEY="your_xquik_key"
   

   This bypasses the Node.js Bird CLI dependency entirely.

Setting Up Exa AI (Optional)

Exa provides semantic search and JS-heavy page fallback:

export EXA_API_KEY="your_exa_key"

---

💡 Usage Examples

Web Research

# Broad search (auto: DDG first, falls back to Exa on error or zero results)
search_web(query="Latest NVIDIA H200 benchmarks")

# Force DDG explicitly
search_web(query="uv package manager", provider="ddg")

# Force Exa explicitly
search_web(query="uv package manager", provider="exa")

# Targeted documentation search
search_web(query="useEffect cleanup", domain="react.dev")

# News with region filter
search_web(query="elections", search_type="news", region="us-en", provider="exa")

# Date-filtered search
search_web(query="uv package manager", time_range="w", provider="auto")

# Deep read a page
fetch_page(url="https://docs.python.org/3/library/os.html")

Technical Analysis

# Track GitHub issues/PRs
search_github(query="uv package manager")

# Get full GitHub issue thread
get_github_issue(url="https://github.com/astral-sh/uv/issues/1")

Community Sentiment

# Reddit discussions
search_reddit(query="Best mechanical keyboards 2024", subreddits=["MechanicalKeyboards"])

# Hacker News technical discourse
search_hackernews(query="MCP server architecture")

# LinkedIn professional search
search_linkedin(query="site reliability engineer", content_type="people")
search_linkedin(query="machine learning startup", content_type="companies")
search_linkedin(query="kubernetes devops", content_type="jobs")
search_linkedin(query="AI agents", content_type="posts")

Academic Research

# arXiv paper search with field prefixes
search_arxiv(query="au:Goodfellow AND cat:cs.LG")
search_arxiv(query="transformer attention", sort_by="submitted_date")

# Wikipedia background research
search_wikipedia(query="Quantum computing")

AI Trends & Forecasting

# Digg AI 1000 - curated clusters from high-signal X accounts
search_digg(query="AI agents")
search_digg(query="Claude Code", depth="deep")

# Techmeme - editorial tech news
search_techmeme(query="Apple WWDC")
search_techmeme(query="AI funding", depth="deep")

# Polymarket - prediction markets with real-money odds
search_polymarket(query="US election 2024")
search_polymarket(query="Fed rate cut")
search_polymarket(query="AI breakthrough", depth="deep")  # expanded query search

---

🏗️ Project Structure

web_search_mcp/
├── server.py              # Entry point: FastMCP init, @mcp.tool registrations
├── search/                # Search engine implementations
│   ├── ddg.py             # DuckDuckGo search + trafilatura page fetch
│   └── exa.py             # Exa SDK search & content fetch (lazy-init client)
├── social/                # Community platform integrations
│   ├── github.py          # GitHub Search API + gh CLI issue rendering
│   ├── hackernews.py      # Algolia HN API + comment enrichment
│   ├── linkedin/          # LinkedIn search via DDG + Jina Reader
│   │   ├── __init__.py    # LinkedIn search tool registration
│   │   └── client.py      # DDG search + Jina Reader enrichment
│   ├── reddit/            # RSS + Shreddit keyless pipeline
│   │   ├── client.py      # HTTP client with RSS parsing
│   │   ├── parsers.py     # RSS/HTML parsers
│   │   └── shreddit.py    # Shreddit comment enrichment
│   └── x.py               # X/Twitter search via Xquik API or vendored Bird CLI
├── tools/                 # Specialized reference utilities
│   ├── arxiv.py           # arXiv paper search (Lucene field prefixes)
│   ├── wikipedia.py       # Wikipedia MediaWiki API
│   ├── digg.py            # Digg AI 1000 clusters via digg-pp-cli
│   ├── techmeme.py        # Techmeme tech news via techmeme-pp-cli
│   └── polymarket.py      # Polymarket prediction markets via Gamma API
├── _config/               # Settings, env vars, rate limits, depth tiers
│   ├── settings.py        # pydantic-settings (EXA_API_KEY, SEARCH_MCP_ prefix)
│   └── limits.py          # Per-platform quick/default/deep limits, timeouts
├── _http/                 # Shared HTTP + SSRF protection
│   └── client.py          # validate_url, http_client, get_json_client
├── _models/               # Pydantic request/response models
│   ├── requests.py        # SearchRequest
│   ├── responses.py       # ErrorResponse, SearchResponse, PageResponse
│   └── types.py           # Depth, ResponseFormat, SearchType, FetchOutputFormat
├── _utils/                # Shared helpers
│   ├── formatting.py      # Markdown formatters, date/epoch utils
│   ├── rate_limiter.py    # Token-bucket rate limiter
│   └── scoring.py         # Relevance scoring
└── vendor/                # Vendored third-party tools
    └── bird-search/       # Node.js CLI for X/Twitter search (fallback when XQUIK_API_KEY unset)

---

🛠️ Tool Implementation Flow

When adding a new tool:

1. Implement logic in the appropriate module (search/, social/, or tools/)
2. Define models in _models/ (request/response types)
3. Register in server.py using @mcp.tool decorator with a clear docstring (serves as the tool’s description for the LLM)

---

📐 Design Decisions

• search-backend-split — Why search_web unifies DuckDuckGo and Exa behind a single provider parameter instead of exposing two separate tools.

---

🧪 Testing

# Run all tests
uv run pytest

# Run a single test file
uv run pytest tests/test_module.py

# Run a specific test
uv run pytest tests/test_module.py::test_function_name

# Run with coverage
uv run pytest --cov=web_search_mcp

---

🔧 Troubleshooting

---

🤝 Contributing

1. Fork the repository.
2. Create a feature branch: git checkout -b feat/my-new-tool
3. Ensure all tests pass: uv run pytest
4. Submit a pull request with a detailed description of the changes.

---

📄 License

This project is licensed under the MIT License.