Crawlab MCP Server

This is a Model Context Protocol (MCP) server for Crawlab, allowing AI applications to interact with Crawlab’s functionality.

Overview

The MCP server provides a standardized way for AI applications to access Crawlab’s features, including:

• Spider management (create, read, update, delete)
• Task management (run, cancel, restart)
• File management (read, write)
• Resource access (spiders, tasks)

Architecture

The MCP Server/Client architecture facilitates communication between AI applications and Crawlab:

graph TB
    User[User] --> Client[MCP Client]
    Client --> LLM[LLM Provider]
    Client  Server[MCP Server]
    Server  Crawlab[Crawlab API]

    subgraph "MCP System"
        Client
        Server
    end

    subgraph "Crawlab System"
        Crawlab
        DB[(Database)]
        Crawlab  DB
    end

    class User,LLM,Crawlab,DB external;
    class Client,Server internal;

    %% Flow annotations
    LLM -.-> |Tool calls| Client
    Client -.-> |Executes tool calls| Server
    Server -.-> |API requests| Crawlab
    Crawlab -.-> |API responses| Server
    Server -.-> |Tool results| Client
    Client -.-> |Human-readable response| User

    classDef external fill:#f9f9f9,stroke:#333,stroke-width:1px;
    classDef internal fill:#d9edf7,stroke:#31708f,stroke-width:1px;

Communication Flow

1. User Query: The user sends a natural language query to the MCP Client
2. LLM Processing: The Client forwards the query to an LLM provider (e.g., Claude, OpenAI)
3. Tool Selection: The LLM identifies necessary tools and generates tool calls
4. Tool Execution: The Client sends tool calls to the MCP Server
5. API Interaction: The Server executes the corresponding Crawlab API requests
6. Response Generation: Results flow back through the Server to the Client to the LLM
7. User Response: The Client delivers the final human-readable response to the user

Installation and Usage

Option 1: Install as a Python package

You can install the MCP server as a Python package, which provides a convenient CLI:

# Install from source
pip install -e .

# Or install from GitHub (when available)
# pip install git+https://github.com/crawlab-team/crawlab-mcp-server.git

After installation, you can use the CLI:

# Start the MCP server
crawlab_mcp-mcp server [--spec PATH_TO_SPEC] [--host HOST] [--port PORT]

# Start the MCP client
crawlab_mcp-mcp client SERVER_URL

Option 2: Running Locally

Prerequisites

• Python 3.8+
• Crawlab instance running and accessible
• API token from Crawlab

Configuration

1. Copy the .env.example file to .env:

   cp .env.example .env
   

2. Edit the .env file with your Crawlab API details:

   CRAWLAB_API_BASE_URL=http://your-crawlab-instance:8080/api
   CRAWLAB_API_TOKEN=your_api_token_here
   

Running Locally

1. Install dependencies:

   pip install -r requirements.txt