이 프로젝트는 크롤링 웹 사이트, 생성 Markdown 문서, 그리고 Cursor와 같은 도구와 통합하도록 설계된 Model Context Protocol (MCP) 서버를 통해 검색할 수 있는 문서를 제공합니다.

특징

• 웹 크롤러 (crawler_cli):
• crawl4ai를 사용하여 주어진 URL에서 시작되는 크롤 웹 사이트.
• 구성 가능한 크롤러 깊이, URL 패턴 (포함 / 제외), 콘텐츠 유형, 기타.
• Markdown 변환 전에 HTML의 선택 청소 (nav 링크, 헤더, 발기자 제거).
• 크롤링 콘텐츠에서 단일, 통합 Markdown 파일 생성.
• ./storage/] 로 출력을 저장합니다.
• MCP 서버 ([T3]]):
• ./storage/ 디렉토리에서 Markdown 파일을로드합니다.
• 파즈 마크다운을 두목에 기초한 semantic chunks로 포즈합니다.
• sentence-transformers] (multi-qa-mpnet-base-dot-v1])를 사용하여 각 펑크에 대한 벡터 embedding를 생성한다. -Caching: 처리된 펑크와 embeddings를 저장하기 위해 캐시 파일(storage/document_chunks_cache.pkl)을 활용합니다.
• 첫 번째 실행: 새 문서의 크롤링 후 초기 서버 시작은 파싱, 펑크, 모든 콘텐츠에 대한 embeddings를 생성해야합니다.

Documentation Crawler & MCP Server

This project provides a toolset to crawl websites, generate Markdown documentation, and make that documentation searchable via a Model Context Protocol (MCP) server, designed for integration with tools like Cursor.

Features

• Web Crawler (crawler_cli):
  • Crawls websites starting from a given URL using crawl4ai.
  • Configurable crawl depth, URL patterns (include/exclude), content types, etc.
  • Optional cleaning of HTML before Markdown conversion (removes nav links, headers, footers).
  • Generates a single, consolidated Markdown file from crawled content.
  • Saves output to ./storage/ by default.
• MCP Server (mcp_server):
  • Loads Markdown files from the ./storage/ directory.
  • Parses Markdown into semantic chunks based on headings.
  • Generates vector embeddings for each chunk using sentence-transformers (multi-qa-mpnet-base-dot-v1).
  • Caching: Utilizes a cache file (storage/document_chunks_cache.pkl) to store processed chunks and embeddings.
    • First Run: The initial server startup after crawling new documents may take some time as it needs to parse, chunk, and generate embeddings for all content.
    • Subsequent Runs: If the cache file exists and the modification times of the source .md files in ./storage/ haven’t changed, the server loads directly from the cache, resulting in much faster startup times.
    • Cache Invalidation: The cache is automatically invalidated and regenerated if any .md file in ./storage/ is modified, added, or removed since the cache was last created.
  • Exposes MCP tools via fastmcp for clients like Cursor:
    • list_documents: Lists available crawled documents.
    • get_document_headings: Retrieves the heading structure for a document.
    • search_documentation: Performs semantic search over document chunks using vector similarity.
• Cursor Integration: Designed to run the MCP server via stdio transport for use within Cursor.

Workflow

1. Crawl: Use the crawler_cli tool to crawl a website and generate a .md file in ./storage/.
2. Run Server: Configure and run the mcp_server (typically managed by an MCP client like Cursor).
3. Load & Embed: The server automatically loads, chunks, and embeds the content from the .md files in ./storage/.
4. Query: Use the MCP client (e.g., Cursor Agent) to interact with the server’s tools (list_documents, search_documentation, etc.) to query the crawled content.

Setup

This project uses uv for dependency management and execution.

1. Install uv: Follow the instructions on the uv website.

2. Clone the repository:

   git clone https://github.com/alizdavoodi/MCPDocSearch.git
   cd MCPDocSearch
   

3. Install dependencies:

   uv sync