为AI代理和AI编码助手提供网页爬取和RAG功能
概览
为AI代理和AI编码助手提供网页爬取和RAG功能 这是与 Crawl4AI 和 Supabase 集成的模型上下文协议（MCP）的强大实现，为 AI 代理和 AI 编码助手提供先进的网页爬取和 RAG 功能。 有了这个MCP服务器，你可以抓取任何东西，然后在任何地方用这些知识来做RAG。 主要目标是将这个MCP服务器带入Archon，同时我将其演变为AI编码助手构建AI代理的知识引擎。Crawl4AI/RAG MCP 服务器的第一个版本很快会得到大幅改进，特别是让它更可配置，这样你可以使用不同的嵌入模型，并用 Ollama 本地运行一切。 把这个GitHub仓库当作一个测试平台，这也是我还没有特别积极处理问题和拉取请求的原因。不过我肯定会这么做，因为我会把它带进Archon V2！
概述（AI 翻译）
Crawl4AI RAG MCP 服务器
为AI代理和AI编码助手提供网页爬取和RAG功能
这是与 Crawl4AI 和 Supabase 集成的模型上下文协议（MCP）的强大实现，为 AI 代理和 AI 编码助手提供先进的网页爬取和 RAG 功能。
有了这个MCP服务器，你可以抓取任何东西，然后在任何地方用这些知识来做RAG。
主要目标是将这个MCP服务器带入Archon，同时我将其演变为AI编码助手构建AI代理的知识引擎。Crawl4AI/RAG MCP 服务器的第一个版本很快会得到大幅改进，特别是让它更可配置，这样你可以使用不同的嵌入模型，并用 Ollama 本地运行一切。
把这个GitHub仓库当作一个测试平台，这也是我还没有特别积极处理问题和拉取请求的原因。不过我肯定会这么做，因为我会把它带进Archon V2！
概述
该MCP服务器提供工具，使AI代理能够爬取网站、将内容存储到矢量数据库（Supabase）中，并对爬取内容执行RAG。
本页由 AI 翻译，技术细节请以原文为准。
README
以下为原文 README 节选，完整内容请查看原文。
Crawl4AI RAG MCP Server
Web Crawling and RAG Capabilities for AI Agents and AI Coding Assistants
A powerful implementation of the Model Context Protocol (MCP) integrated with Crawl4AI and Supabase for providing AI agents and AI coding assistants with advanced web crawling and RAG capabilities.
With this MCP server, you can scrape anything and then use that knowledge anywhere for RAG.
The primary goal is to bring this MCP server into Archon as I evolve it to be more of a knowledge engine for AI coding assistants to build AI agents. This first version of the Crawl4AI/RAG MCP server will be improved upon greatly soon, especially making it more configurable so you can use different embedding models and run everything locally with Ollama.
Consider this GitHub repository a testbed, hence why I haven’t been super actively address issues and pull requests yet. I certainly will though as I bring this into Archon V2!
Overview
This MCP server provides tools that enable AI agents to crawl websites, store content in a vector database (Supabase), and perform RAG over the crawled content. It follows the best practices for building MCP servers based on the Mem0 MCP server template I provided on my channel previously.
The server includes several advanced RAG strategies that can be enabled to enhance retrieval quality:
- Contextual Embeddings for enriched semantic understanding
- Hybrid Search combining vector and keyword search
- Agentic RAG for specialized code example extraction
- Reranking for improved result relevance using cross-encoder models
- Knowledge Graph for AI hallucination detection and repository code analysis
See the Configuration section below for details on how to enable and configure these strategies.
Vision
The Crawl4AI RAG MCP server is just the beginning. Here’s where we’re headed:
-
Integration with Archon: Building this system directly into Archon to create a comprehensive knowledge engine for AI coding assistants to build better AI agents.
-
Multiple Embedding Models: Expanding beyond OpenAI to support a variety of embedding models, including the ability to run everything locally with Ollama for complete control and privacy.
-
Advanced RAG Strategies: Implementing sophisticated retrieval techniques like contextual retrieval, late chunking, and others to move beyond basic “naive lookups” and significantly enhance the power and precision of the RAG system, especially as it integrates with Archon.
安装
docker run --rm -i -e TRANSPORT -e OPENAI_API_KEY -e SUPABASE_URL -e SUPABASE_SERVICE_KEY -e USE_KNOWLEDGE_GRAPH -e NEO4J_URI -e NEO4J_USER -e NEO4J_PASSWORD mcp/crawl4ai配置
{
"mcpServers": {
"crawl4ai-rag": {
"transport": "sse",
"url": "http://localhost:8051/sse"
}
}
}