该项目提供了一个MCP（模型上下文协议）服务器，利用crawl4ai库执行网页抓取和智能内容提取任务。它允许AI代理（如Claude，或用LangChain/LangGraph构建的代理）与网页交互，检索内容，搜索特定文本，并基于自然语言指令执行基于LLM的提取。

该服务器使用：

• FastMCP： 用于创建MCP服务器端点。
• crawl4ai： 用于核心网页爬行和提取逻辑。
• dotenv： 通过 .env 文件管理 API 密钥。
• （可选）Docker： 用于容器化部署，捆绑Python和依赖。

特色

• 公开用于网页交互的MCP工具：
• scrape_url：获取网页的完整内容，采用Markdown格式。
• extract_text_by_query：根据查询查找页面上的具体文本片段。
• smart_extract：使用大型语言模型（目前为Google Gemini）根据指令提取结构化信息。
• 可通过环境变量（API 密钥）配置。
• 包含 Docker 配置（[Dockerfile），便于自包含部署。
• 默认通过8002端口的服务器发送事件（SSE）通信。

Crawl4AI Web Scraper MCP Server

This project provides an MCP (Model Context Protocol) server that uses the crawl4ai library to perform web scraping and intelligent content extraction tasks. It allows AI agents (like Claude, or agents built with LangChain/LangGraph) to interact with web pages, retrieve content, search for specific text, and perform LLM-based extraction based on natural language instructions.

This server uses:

• FastMCP: For creating the MCP server endpoint.
• crawl4ai: For the core web crawling and extraction logic.
• dotenv: For managing API keys via a .env file.
• (Optional) Docker: For containerized deployment, bundling Python and dependencies.

Features

• Exposes MCP tools for web interaction:
  • scrape_url: Get the full content of a webpage in Markdown format.
  • extract_text_by_query: Find specific text snippets on a page based on a query.
  • smart_extract: Use an LLM (currently Google Gemini) to extract structured information based on instructions.
• Configurable via environment variables (API keys).
• Includes Docker configuration (Dockerfile) for easy, self-contained deployment.
• Communicates over Server-Sent Events (SSE) on port 8002 by default.

Exposed MCP Tools

scrape_url

Scrape a webpage and return its content in Markdown format.

Arguments:

• url (str, required): The URL of the webpage to scrape.

Returns:

• (str): The webpage content in Markdown format, or an error message.

extract_text_by_query

Extract relevant text snippets from a webpage that contain a specific search query. Returns up to the first 5 matches found.

Arguments:

• url (str, required): The URL of the webpage to search within.
• query (str, required): The text query to search for (case-insensitive).
• context_size (int, optional): The number of characters to include before and after the matched query text in each snippet. Defaults to 300.

Returns:

• (str): A formatted string containing the found text snippets or a message indicating no matches were found, or an error message.

smart_extract

Intelligently extract specific information from a webpage using the configured LLM (currently requires Google Gemini API key) based on a natural language instruction.

Arguments:

• url (str, required): The URL of the webpage to analyze and extract from.