이 프로젝트는 crawl4ai 라이브러리를 사용하여 MCP (Model Context Protocol) 서버를 제공하여 웹 스크랩 및 지능형 콘텐츠 추출 작업을 수행합니다. 그것은 AI 에이전트 (클래드와 같은, 또는 에이전트는 LangChain / LangGraph와 함께 구축) 웹 페이지와 상호 작용, 특정 텍스트 검색, 자연 언어 지침을 기반으로 LLM 기반 추출 수행.

이 서버는:

• FastMCP: MCP 서버 엔드포인트 만들기
• ** 크롤러4ai:** 핵심 웹 크롤링 및 추출 논리를 위해.
• ** dotenv:** .env] 파일을 통해 API 키 관리
• ** (선택) 도커: ** 컨테이너화된 배포를 위해, Python 및 의존성을 묶습니다.

특징

• 웹 상호 작용을 위한 MCP 공구를 노출하십시오:
• scrape_url: Markdown 형식의 웹 페이지의 전체 내용을 가져옵니다.
• extract_text_by_query: 자주 묻는 질문(FAQ)
• smart_extract: LLM (현재 Google Gemini)을 사용하여 지시에 따라 구조화된 정보를 추출합니다.
• 환경 변수 (API 키)를 통해 구성 가능.
• Docker 구성(Dockerfile])을 쉽고, 자체 유지된 배포를 포함합니다.
• 기본적으로 포트 8002에 Server-Sent Events (SSE)를 통해 통신합니다.

Crawl4AI Web Scraper MCP Server

This project provides an MCP (Model Context Protocol) server that uses the crawl4ai library to perform web scraping and intelligent content extraction tasks. It allows AI agents (like Claude, or agents built with LangChain/LangGraph) to interact with web pages, retrieve content, search for specific text, and perform LLM-based extraction based on natural language instructions.

This server uses:

• FastMCP: For creating the MCP server endpoint.
• crawl4ai: For the core web crawling and extraction logic.
• dotenv: For managing API keys via a .env file.
• (Optional) Docker: For containerized deployment, bundling Python and dependencies.

Features

• Exposes MCP tools for web interaction:
  • scrape_url: Get the full content of a webpage in Markdown format.
  • extract_text_by_query: Find specific text snippets on a page based on a query.
  • smart_extract: Use an LLM (currently Google Gemini) to extract structured information based on instructions.
• Configurable via environment variables (API keys).
• Includes Docker configuration (Dockerfile) for easy, self-contained deployment.
• Communicates over Server-Sent Events (SSE) on port 8002 by default.

Exposed MCP Tools

scrape_url

Scrape a webpage and return its content in Markdown format.

Arguments:

• url (str, required): The URL of the webpage to scrape.

Returns:

• (str): The webpage content in Markdown format, or an error message.

extract_text_by_query

Extract relevant text snippets from a webpage that contain a specific search query. Returns up to the first 5 matches found.

Arguments:

• url (str, required): The URL of the webpage to search within.
• query (str, required): The text query to search for (case-insensitive).
• context_size (int, optional): The number of characters to include before and after the matched query text in each snippet. Defaults to 300.

Returns:

• (str): A formatted string containing the found text snippets or a message indicating no matches were found, or an error message.

smart_extract

Intelligently extract specific information from a webpage using the configured LLM (currently requires Google Gemini API key) based on a natural language instruction.

Arguments:

• url (str, required): The URL of the webpage to analyze and extract from.