tokidoo/crawl4ai-rag-mcp

Crawl4AI+SearXNG MCPサーバー

AIエージェントとAIコーディングアシスタントのWebクロール、検索、RAG機能

([[https://github.com/coleam00/mcp-crawl4ai-rag).から始まる) SearXNGの統合とバッチスクレイプと処理機能を追加しました。

モデルコンテクストプロトコル(MCP)、Crawl4AI、SearXNG、Supabaseを組み合わせて、AIエージェントとコーディングアシスタントを完全なWebsearch、クローリング、およびRAG機能で提供できるようにします。

** 1 つのコマンドで完全なスタック**: [docker compose up -d]ですべてをデプロイする - Python の設定、依存関係なし、外部サービスは必要ありません。

は、スマートRAGと伝統的なスクレイピング

従来のスクレイピング(火炎など)とは異なり、生のコンテンツをダンプし、LMコンテキストウィンドウを圧倒するこのソリューションは、**intelligent RAG(Retrieval Augmented generation)**を使用します。

• ** 参照 関連するコンテンツだけを抽出** semantic類似性検索を使用して -** ⚡ コンテキストのオーバーフローを防止** 焦点を絞った情報を返す -** ターゲットを絞った知識でAIの応答を強化する**
• よりよいLMの性能のためのコンテキストの効率を維持して下さい

柔軟な出力オプション:

🐳 Crawl4AI+SearXNG MCP Server

Web Crawling, Search and RAG Capabilities for AI Agents and AI Coding Assistants

(FORKED FROM https://github.com/coleam00/mcp-crawl4ai-rag). Added SearXNG integration and batch scrape and processing capabilities.

A self-contained Docker solution that combines the Model Context Protocol (MCP), Crawl4AI, SearXNG, and Supabase to provide AI agents and coding assistants with complete web search, crawling, and RAG capabilities.

🚀 Complete Stack in One Command: Deploy everything with docker compose up -d - no Python setup, no dependencies, no external services required.

🎯 Smart RAG vs Traditional Scraping

Unlike traditional scraping (such as Firecrawl) that dumps raw content and overwhelms LLM context windows, this solution uses intelligent RAG (Retrieval Augmented Generation) to:

• 🔍 Extract only relevant content using semantic similarity search
• ⚡ Prevent context overflow by returning focused, pertinent information
• 🧠 Enhance AI responses with precisely targeted knowledge
• 📊 Maintain context efficiency for better LLM performance

Flexible Output Options:

• RAG Mode (default): Returns semantically relevant chunks with similarity scores
• Raw Markdown Mode: Full content extraction when complete context is needed
• Hybrid Search: Combines semantic and keyword search for comprehensive results

💡 Key Benefits

• 🔧 Zero Configuration: Pre-configured SearXNG instance included
• 🐳 Docker-Only: No Python environment setup required
• 🔍 Integrated Search: Built-in SearXNG for private, fast search
• ⚡ Production Ready: HTTPS, security, and monitoring included
• 🎯 AI-Optimized: RAG strategies built for coding assistants

Overview

This Docker-based MCP server provides a complete web intelligence stack that enables AI agents to:

• Search the web using the integrated SearXNG instance
• Crawl and scrape websites with advanced content extraction
• Store content in vector databases with intelligent chunking
• Perform RAG queries with multiple enhancement strategies

Advanced RAG Strategies Available:

• Contextual Embeddings for enriched semantic understanding
• Hybrid Search combining vector and keyword search
• Agentic RAG for specialized code example extraction
• Reranking for improved result relevance using cross-encoder models
• Knowledge Graph for AI hallucination detection and repository code analysis

See the Configuration section below for details on how to enable and configure these strategies.

Features

• Smart URL Detection: Automatically detects and handles different URL types (regular webpages, sitemaps, text files)