77777r7/octocrawl
Developer toolsこのページは AI により自動翻訳されています。技術的な詳細は原文をご確認ください。 原文を見る
Octocrawl: readable Markdown and checked fields from public web pages, for AI agents, RAG and research. Failures come back with a reason. Self-host with REST, SDK or MCP.
概要
One link. Web data, ready. Markdown and data from web pages, for AI agents and RAG. Every result records how it was read. Every failure says why. Website · Docs · Connect MCP · Blog · Changelog Octocrawl is an open-source web scraping and crawling toolkit for AI agents, RAG and research. Collect documentation for a knowledge base, extract product details for comparison, or turn web tables and listings into data for analysis. Each page comes with a source record. Try a page in your browser, or connect your agent through hosted MCP or the REST API. No installation or API key is needed to try the hosted service. Hosted Octocrawl offers ; keyless requests use HTTP, and an account key enables the browser lane. Run locally for crawl, batch, page actions and Monitors; see Running Octocrawl for setup and self-hosting options. The web preview is a single-page trial, with five pages a day and up to 20 named fields.
README
以下は原文 README の抜粋です。原文を見る。
Octocrawl
One link. Web data, ready. Markdown and data from web pages, for AI agents and RAG. Fewer tokens than raw HTML. Every result records how it was read.
Website · Docs · Connect MCP · Blog · Changelog
Scrape pages. Crawl websites. Turn web content into usable data.
Octocrawl is an open-source web scraping and crawling toolkit for AI agents, RAG and research. Collect documentation for a knowledge base, extract product details for comparison, or turn web tables and listings into data for analysis. Each page comes with a source record.
Try a page in your browser, or connect your agent through hosted MCP or the REST API. No installation or API key is needed to try the hosted service.
What can you do with Octocrawl?
| Capability | What you get |
|---|---|
| Scrape a page | Readable Markdown for LLM context, plus HTML, links, image URLs and page metadata. Request a screenshot on the browser lane. |
| Map a website | Discover URLs from sitemaps and page links, and filter them by a search term to find the pages you need. |
| Crawl a site or section | Follow links from a starting URL and collect each page’s content, with path filters, depth and page limits. |
インストール
npx octocrawl scrape https://example.com --markdown