77777r7/octocrawl
Developer toolsЭта страница переведена автоматически. Технические детали смотрите в оригинале. Открыть оригинал
Octocrawl: readable Markdown and checked fields from public web pages, for AI agents, RAG and research. Failures come back with a reason. Self-host with REST, SDK or MCP.
Обзор
One link. Web data, ready. Markdown and data from web pages, for AI agents and RAG. Every result records how it was read. Every failure says why. Website · Docs · Connect MCP · Blog · Changelog Octocrawl is an open-source web scraping and crawling toolkit for AI agents, RAG and research. Collect documentation for a knowledge base, extract product details for comparison, or turn web tables and listings into data for analysis. Each page comes with a source record. Try a page in your browser, or connect your agent through hosted MCP or the REST API. No installation or API key is needed to try the hosted service. Hosted Octocrawl offers ; keyless requests use HTTP, and an account key enables the browser lane. Run locally for crawl, batch, page actions and Monitors; see Running Octocrawl for setup and self-hosting options. The web preview is a single-page trial, with five pages a day and up to 20 named fields.
README
Показан фрагмент оригинального README. Открыть оригинал.
Octocrawl
One link. Web data, ready. Markdown and data from web pages, for AI agents and RAG. Fewer tokens than raw HTML. Every result records how it was read.
Website · Docs · Connect MCP · Blog · Changelog
Scrape pages. Crawl websites. Turn web content into usable data.
Octocrawl is an open-source web scraping and crawling toolkit for AI agents, RAG and research. Collect documentation for a knowledge base, extract product details for comparison, or turn web tables and listings into data for analysis. Each page comes with a source record.
Try a page in your browser, or connect your agent through hosted MCP or the REST API. No installation or API key is needed to try the hosted service.
What can you do with Octocrawl?
| Capability | What you get |
|---|---|
| Scrape a page | Readable Markdown for LLM context, plus HTML, links, image URLs and page metadata. Request a screenshot on the browser lane. |
| Map a website | Discover URLs from sitemaps and page links, and filter them by a search term to find the pages you need. |
| Crawl a site or section | Follow links from a starting URL and collect each page’s content, with path filters, depth and page limits. |
Установка
npx octocrawl scrape https://example.com --markdown