pazyork/crawl4ai-mcp

Developer tools

このページは AI により自動翻訳されています。技術的な詳細は原文をご確認ください。原文を見る

3 stars 0 forks 品質 63 トレンド 8NOASSERTION

Repository Install

A minimal MCP server built on crawl4ai for LLM-friendly web extraction

概要

Two tools: fetch real pages with Playwright + Crawl4AI, or search across 7 engines with automatic fallback. - Check Python version: python3 --version (must be 3.10-3.13) - Upgrade pip: python3 -m pip install --upgrade pip - Use mirror (China): export PLAYWRIGHT_DOWNLOAD_HOST=https://npmmirror.com/mirrors/playwright/ - Then: python3 -m playwright install chromium Most generic “web fetch” tools either fail on JS-heavy pages or return too much boilerplate. This project focuses on four things: - : usable even with zero model config - : easier for agents, easier to maintain - : proxy / cookies / persistent profile are first-class - : full markdown outputs can be saved and inspected page by page - With llm_instruction, the prompt is and biased toward preserving original lines. - Without llm_instruction, the model does a more generic “clean readable markdown” pass. - If the LLM call fails for any reason, the tool returns the original non-LLM extraction plus llm_used=false and llm_error.

README

以下は原文 README の抜粋です。原文を見る。

crawl4ai-mcp

Quick entry

Audience	Read this
Human developer	README.zh-CN.md / README.md
Living in the AI era, delegating your remaining sanity to an agent	README_AGENT.md

At a glance

Item	Reality in this repo
MCP tools	2 tools: `fetch_urls` + `search_web`
Single-page fetch	`urls: ["https://example.com"]`
Web search	`search_web(query="...", engine="auto")` — 7 engines, auto fallback
Search engines	DuckDuckGo · Bing · Google · Yandex · Sogou · 360Search · Baidu
Output	`title + content + links + blocked + llm_used/llm_error`
Non-LLM mode	First-class, default, usable without any model
LLM mode	Off by default. Enabled only with `use_llm=true` + optional `llm_instruction`
Fallback	Missing/failed LLM call automatically falls back to non-LLM result
Anti-bot realism	proxy / cookies / persistent profile / randomized browser behavior
License	AGPL-3.0-or-later

How it works

Fetch flow:

---
config:
  theme: base
  themeVariables:
    primaryColor: "#FAF9F5"
    primaryTextColor: "#1A1A1A"
    primaryBorderColor: "#D97757"
    lineColor: "#8B5E3C"
    secondaryColor: "#F5F1E8"
    tertiaryColor: "#FAF9F5"
    fontFamily: "ui-sans-serif, system-ui, -apple-system, sans-serif"
    fontSize: "14px"
---
flowchart LR
    A[URL list] --> B[Playwright + Crawl4AI]
    B --> C{Fast path enough?}
    C -- Yes --> D[Markdown / HTML]
    C -- No --> E[Stronger fallback]
    E --> D
    D --> F{use_llm?}
    F -- No --> G[Return result]
    F -- Yes --> H[OpenAI-compatible cleanup]
    H --> I{LLM success?}
    I -- Yes --> J[Return enhanced result]
    I -- No --> G

Search flow:

---
config:
  theme: base
  themeVariables:
    primaryColor: "#FAF9F5"
    primaryTextColor: "#1A1A1A"
    primaryBorderColor: "#D97757"

GitHub で README 全文を見る

インストール

This server does not publish a one-line install command.

Open the repository installation guide MCP 初心者？mcpskill.org のガイドから始めましょう →

設定

{
  "mcpServers": {
    "crawl4ai": {
      "command": "crawl4ai-mcp",
      "env": {
        "CRAWL4AI_MCP_HEADLESS": "true",
        "CRAWL4AI_MCP_PROXY": "127.0.0.1:7890",
        "CRAWL4AI_MCP_NAVIGATION_TIMEOUT_MS": "30000",
        "CRAWL4AI_MCP_WAIT_UNTIL": "load",

        "OPENAI_BASE_URL": "https://your-openai-compatible-host",
        "OPENAI_API_KEY": "your-api-key",
        "OPENAI_MODEL": "your-model-name"
      }
    }
  }
}