sisig-ai/doctor

🩺 医生

一个用于发现、爬取和索引网站的工具，以MCP服务器形式公开给LLM代理，实现更好、更及时的推理和代码生成。

---

🔍 概述

医生提供了完整的堆栈：

• 使用 crawl4ai 进行爬取网页，并带有层级跟踪
• 使用LangChain的分块文本
• 通过 litellm 创建 OpenAI 嵌入
• 支持向量搜索的DuckDB数据存储
• 通过FastAPI网络服务暴露搜索功能
• 通过MCP服务器向LLM提供这些功能
• 使用分层站点地图导航爬行站点

---

🏗️ 核心基础设施

🗄️ 鸭子数据库

• 用于存储文档数据和嵌入的数据库，具备矢量搜索功能
• 由统一数据库类管理

📨 雷迪斯

• 异步任务处理的消息代理

🕸️ 爬行工人

• 进程爬取作业
• 块 文本
• 创建嵌入

🌐 网络服务器

• FastAPI服务，暴露端点
• 获取、搜索和查看数据
• 暴露MCP服务器

---

💻 布置

⚙️ 先决条件

• Docker 和 Docker 组合
• Python 3.10+
• uv（Python 包管理器）
• OpenAI API 密钥

---

🔍 Overview

Doctor provides a complete stack for:

• Crawling web pages using crawl4ai with hierarchy tracking
• Chunking text with LangChain
• Creating embeddings with OpenAI via litellm
• Storing data in DuckDB with vector search support
• Exposing search functionality via a FastAPI web service
• Making these capabilities available to LLMs through an MCP server
• Navigating crawled sites with hierarchical site maps

---

🏗️ Core Infrastructure

🗄️ DuckDB

• Database for storing document data and embeddings with vector search capabilities
• Managed by unified Database class

📨 Redis

• Message broker for asynchronous task processing

🕸️ Crawl Worker

• Processes crawl jobs
• Chunks text
• Creates embeddings

🌐 Web Server

• FastAPI service exposing endpoints
• Fetching, searching, and viewing data
• Exposing the MCP server

---

💻 Setup

⚙️ Prerequisites

• Docker and Docker Compose
• Python 3.10+
• uv (Python package manager)
• OpenAI API key

📦 Installation

1. Clone this repository
2. Set up environment variables:

   export OPENAI_API_KEY=your-openai-key
   

3. Run the stack:

   docker compose up
   

---

👁 Usage

1. Go to http://localhost:9111/docs to see the OpenAPI docs
2. Look for the /fetch_url endpoint and start a crawl job by providing a URL
3. Use /job_progress to see the current job status
4. Configure your editor to use http://localhost:9111/mcp as an MCP server

---

☁️ Web API

Core Endpoints

• POST /fetch_url: Start crawling a URL
• GET /search_docs: Search indexed documents
• GET /job_progress: Check crawl job progress
• GET /list_doc_pages: List indexed pages
• GET /get_doc_page: Get full text of a page

Site Map Feature

The Maps feature provides a hierarchical view of crawled websites, making it easy to navigate and explore the structure of indexed sites.

Endpoints:

• GET /map: View an index of all crawled sites
• GET /map/site/{root_page_id}: View the hierarchical tree structure of a specific site