77777r7/octocrawl
Developer tools이 페이지는 AI로 자동 번역되었습니다. 기술적인 세부 사항은 원문을 확인하세요. 원문 보기
Octocrawl: readable Markdown and checked fields from public web pages, for AI agents, RAG and research. Failures come back with a reason. Self-host with REST, SDK or MCP.
개요
One link. Web data, ready. Markdown and data from web pages, for AI agents and RAG. Every result records how it was read. Every failure says why. Website · Docs · Connect MCP · Blog · Changelog Octocrawl is an open-source web scraping and crawling toolkit for AI agents, RAG and research. Collect documentation for a knowledge base, extract product details for comparison, or turn web tables and listings into data for analysis. Each page comes with a source record. Try a page in your browser, or connect your agent through hosted MCP or the REST API. No installation or API key is needed to try the hosted service. Hosted Octocrawl offers ; keyless requests use HTTP, and an account key enables the browser lane. Run locally for crawl, batch, page actions and Monitors; see Running Octocrawl for setup and self-hosting options. The web preview is a single-page trial, with five pages a day and up to 20 named fields.
README
원문 README의 일부입니다. 원문 보기.
Octocrawl
One link. Web data, ready. Markdown and data from web pages, for AI agents and RAG. Fewer tokens than raw HTML. Every result records how it was read.
Website · Docs · Connect MCP · Blog · Changelog
Scrape pages. Crawl websites. Turn web content into usable data.
Octocrawl is an open-source web scraping and crawling toolkit for AI agents, RAG and research. Collect documentation for a knowledge base, extract product details for comparison, or turn web tables and listings into data for analysis. Each page comes with a source record.
Try a page in your browser, or connect your agent through hosted MCP or the REST API. No installation or API key is needed to try the hosted service.
What can you do with Octocrawl?
| Capability | What you get |
|---|---|
| Scrape a page | Readable Markdown for LLM context, plus HTML, links, image URLs and page metadata. Request a screenshot on the browser lane. |
| Map a website | Discover URLs from sitemaps and page links, and filter them by a search term to find the pages you need. |
| Crawl a site or section | Follow links from a starting URL and collect each page’s content, with path filters, depth and page limits. |
설치
npx octocrawl scrape https://example.com --markdown