sanggonboy/pyrecrawl

Эта страница переведена автоматически. Технические детали смотрите в оригинале. Открыть оригинал

22 stars 2 forks Качество 82 Тренд 18MIT

Web browsing superpowers for AI agents - one MCP server to scrape, crawl, extract, map, search, batch, research & monitor the web. Self-hosted, no API keys. Free Firecrawl alternative.

Обзор

Scrape, extract, crawl, map, and search — self-hosted, no API keys, no rate limits, no subscription. PyreCrawl speaks (Model Context Protocol), the standard tool interface for Claude, Cursor, VS Code, Codex, OpenCode, Hermes, and any MCP-compatible agent. A always picks the cheapest method that succeeds: (read-only state without a tool call): pyrecrawl://cache/stats · pyrecrawl://sessions · pyrecrawl://monitors (ready-made playbooks): research(topic) · rag_ingest(site) · watch_page(url) prefer options: "auto" (default ladder) · "fast" (HTTP only) · "stealth" (CF bypass) · "llm" (deep processing). UV is a fast Python package manager that handles Python itself — no need to install Python separately. Get it once: Then run PyreCrawl directly — no venv, no pip install, no Python download:

README

Показан фрагмент оригинального README. Открыть оригинал.

🔥 PyreCrawl — Web Browsing Superpowers for Your AI Agent

One command gives any AI agent the whole web. Scrape, extract, crawl, map, and search — self-hosted, no API keys, no rate limits, no subscription.

PyreCrawl speaks MCP (Model Context Protocol), the standard tool interface for Claude, Cursor, VS Code, Codex, OpenCode, Hermes, and any MCP-compatible agent.

A smart auto-fallback ladder always picks the cheapest method that succeeds:

fast HTTP
    │  (403/503/Cloudflare challenge or empty body)
    ▼
stealth browser (real Chromium + Cloudflare solver)
    │  (still blocked, or the page needs full JS rendering)
    ▼
deep processing (LLM-ready markdown, citations, structured extraction)

⚡ Tools exposed

Tool	What it does
`scrape(url, prefer="auto")`	Single URL → LLM-ready markdown
`extract(url, schema)`	Scrape + structured extraction (JsonCss schema)
`map_site(root, include_pattern=None, limit=200)`	Enumerate all internal URLs
`crawl(root, max_pages=5, prefer="auto", include_paths=None, exclude_paths=None, max_depth=0)`	Multi-page crawl with path filters + true BFS depth
`document(url)`	PDF/DOCX/PPTX → markdown (no browser, optional `[docs]` extras)
`search(query, limit=10)`	Web search via DuckDuckGo HTML (no API key)
`search_papers(query, limit=8, source="arxiv", category=None)`	Academic search via arXiv + Crossref (no API key) — feed `pdf_url` into `document`
`batch_scrape(urls[], ...)`	Many URLs in ONE call — parallel, deduped, cache-aware
`deep_research(query, limit=5, scrape_top=3)`	Search → evidence pack with [n] citations (no LLM synthesis — your agent does that)
`monitor(url, action, css_selector=None)`	Change detection with persisted snapshots + unified diff
`session(session, action, ...)`	Persistent browser session (cookies kept) — login walls, multi-step flows, screenshots
`cache(action)`	Inspect/clear/enable/disable the HTTP response cache
`health()`	Versions + import sanity check

Открыть полный README на GitHub

Установка

uvx --from pyrecrawl pyrecrawl serve

Новичок в MCP? Начните с руководства на mcpskill.org →

Конфигурация

{
  "mcpServers": {
    "pyrecrawl": {
      "command": "uvx",
      "args": ["--from", "pyrecrawl", "pyrecrawl", "serve"]
    }
  }
}