nirholas/extract-llms-docs

34 stars 0 forks Качество 90 Тренд 90

Extract documentation for AI agents from any site with llms.txt support. Features MCP server, REST API, batch processing, and multiple export formats.

Обзор

Extract llms.txt documentation and install.md instructions from any website for AI agents, LLMs, and automation workflows. llm.energy llms-full-txt.vercel.app llm.energy is a web application and MCP server that fetches, parses, and organizes documentation from websites implementing the llms.txt and install.md standards. It transforms raw documentation into structured, agent-ready formats optimized for large language models, AI assistants, and developer tooling. Generate LLM-executable installation instructions from any source: 1. : Analyzes README, package.json/pyproject.toml/Cargo.toml, GitHub Actions, and releases 2. : Scrapes documentation pages, detects platforms (Mintlify, Docusaurus, GitBook, etc.) 3. : Uses Claude to generate a properly formatted install.md Visit llm.energy to use the hosted version. Add to your MCP client configuration (Claude Desktop, Cursor, etc.): See MCP Server Documentation for detailed setup. POST /api/extract - Extract documentation from a URL

README

llm.energy

Extract llms.txt documentation and install.md instructions from any website for AI agents, LLMs, and automation workflows.

llm.energy

llms-full-txt.vercel.app

📖 Overview

llm.energy is a web application and MCP server that fetches, parses, and organizes documentation from websites implementing the llms.txt and install.md standards. It transforms raw documentation into structured, agent-ready formats optimized for large language models, AI assistants, and developer tooling.

Standard	Description	Learn More
llms.txt	Machine-readable documentation for AI systems	llmstxt.org
install.md	LLM-executable installation instructions	installmd.org

✨ Key Features

Feature	Description
🔍 Smart Detection	Automatically finds `llms.txt`, `llms-full.txt`, and `install.md`
📄 Organized Output	Splits content into individual markdown files by section
🤖 Agent-Ready	Includes `AGENT-GUIDE.md` optimized for AI assistants
📦 Multiple Formats	Export as Markdown, JSON, YAML, or ZIP archive
🔌 MCP Server	Integrate with Claude Desktop, Cursor, and more
⚡ Batch Processing	Process multiple URLs simultaneously
📚 Site Directory	Browse 19+ curated llms.txt-enabled websites
✏️ llms.txt Generator	Create your own llms.txt files with a guided wizard
🚀 install.md Generator	AI-powered: generate from GitHub repos, docs URLs, or manually

🎯 Use Cases

─────────────────────────────────────────────────────────────────
                                                                 
   📝 Feed docs to AI coding assistants (Cursor, Windsurf)       
   🤖 Build context-aware AI agents with up-to-date docs         
   🔄 Create documentation pipelines for RAG systems            
   📦 Aggregate docs from multiple sources automatically        
   ✏️  Generate llms.txt/install.md for your own projects       
   🚀 Auto-generate install.md from any GitHub repo             
                                                                 
─────────────────────────────────────────────────────────────────

🚀 install.md Generator

Generate LLM-executable installation instructions from any source:

Mode	Description	Use Case
From GitHub	Analyze any public GitHub repo and generate install.md	Perfect for creating install.md for existing projects
From URL	Extract from any documentation page	Convert existing docs to install.md format
Manual	Build from scratch with guided wizard	Full control over every detail

How It Works

GitHub Mode: Analyzes README, package.json/pyproject.toml/Cargo.toml, GitHub Actions, and releases
URL Mode: Scrapes documentation pages, detects platforms (Mintlify, Docusaurus, GitBook, etc.)
AI Synthesis: Uses Claude to generate a properly formatted install.md

Quick Example

# Generate install.md for any GitHub project
curl -X POST https://llm-energy-lp642k3kpa-uc.a.run.app/api/generate-install \
  -H "Content-Type: application/json" \
  -d '{"url": "https://github.com/anthropics/anthropic-sdk-python", "type": "github"}'

Visit llm.energy/install-generator to use the web interface.

📚 Documentation

Full documentation is available at llm.energy/docs

🚀 Installation

Web Application

Visit llm.energy to use the hosted version.

Local Development

# Clone the repository
git clone https://github.com/nirholas/extract-llms-docs.git
cd extract-llms-docs

# Install dependencies
pnpm install

# Start development server
pnpm dev

🌐 The application runs on http://localhost:3001

MCP Server

🔌 API Reference

📖 Full API Reference →

🤖 MCP Server Tools

The MCP server exposes the following tools for AI agents:

Tool	Description
`extract_docs`	Extract documentation from a URL with llms.txt support
`validate_url`	Check if a URL has llms.txt available
`verify_llms_txt`	Verify llms.txt exists and get file info
`discover_documentation_urls`	Find documentation URLs for a domain
`list_sites`	Get directory of known llms.txt-enabled sites
`search_sites`	Search the site directory by category or keyword

🏗️ Architecture

┌──────────────────────────────────────────────────────────────────┐
│                        llm.energy                                │
├──────────────────────────────────────────────────────────────────┤
│                                                                  │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐               │
│  │   Web App   │  │  REST API   │  │ MCP Server  │               │
│  │  (Next.js)  │  │  /api/*     │  │   (stdio)   │               │
│  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘               │
│         │                │                │                      │
│         └────────────────┼────────────────┘                      │
│                          │                                       │
│                   ┌──────▼──────┐                                │
│                   │    Core     │                                │
│                   │  - Parser   │                                │
│                   │  - Extractor│                                │
│                   │  - Cache    │                                │
│                   └──────┬──────┘                                │
│                          │                                       │
│         ┌────────────────┼────────────────┐                      │
│         ▼                ▼                ▼                      │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐               │
│  │  llms.txt   │  │ install.md  │  │   Sitemap   │               │
│  └─────────────┘  └─────────────┘  └─────────────┘               │
│                                                                  │
└──────────────────────────────────────────────────────────────────┘

🛠️ Technology Stack

⚙️ Configuration

All environment variables are optional — the app runs without them. Copy .env.example to .env.local and fill in only what you need:

cp .env.example .env.local

Variable	Used by	Default	Purpose
`ADMIN_KEY`	`/api/cache`	unset	Value the `x-admin-key` header must match for cache management. Set this in any deployment — the prune endpoint otherwise falls back to a well-known default.
`ANTHROPIC_API_KEY`	`/api/generate-install`	unset	Enables Claude-backed generation of `install.md`. The endpoint returns a configuration error without it.
`VERIFY_TIMEOUT_MS`	`/api/sites/verify`	`10000`	Per-site fetch timeout, in milliseconds.
`VERIFY_CACHE_TTL_MS`	`/api/sites/verify`	`300000`	How long verification results are cached, in milliseconds.
`HEALTH_TIMEOUT_MS`	`/api/sites/health`	`8000`	Timeout for site health probes, in milliseconds.

💻 Development

pnpm dev          # 🚀 Start development server
pnpm build        # 📦 Production build
pnpm start        # ▶️  Start production server
pnpm test         # 🧪 Run tests (163 tests)
pnpm test:coverage # 📊 Tests with coverage report
pnpm typecheck    # ✅ TypeScript validation
pnpm lint         # 🔍 ESLint check

Project	Description
llms.txt Standard	Machine-readable documentation format
install.md Standard	LLM-executable installation format
Model Context Protocol	Protocol for AI tool integration
MCP Servers Directory	Community MCP servers

🌐 Sites with llms.txt Support

Browse 19+ curated websites with verified llms.txt support at llm.energy/directory

Featured sites include:

🤖 Anthropic Documentation
⚡ Vercel Documentation
💳 Stripe API Reference
🗄️ Supabase Docs
📘 Mintlify Documentation
…and more!

🤝 Contributing

Contributions are welcome! Please follow these steps:

🍴 Fork the repository
🌿 Create a feature branch (git checkout -b feature/amazing-feature)
💾 Commit your changes (git commit -m 'Add amazing feature')
📤 Push to the branch (git push origin feature/amazing-feature)
🔀 Open a Pull Request

Adding a site to the directory? Edit src/data/sites.ts and submit a PR.

📄 License

🔗 Links

🌐 Website • 📚 Documentation • 💻 GitHub • 📦 NPM

👤 Author

Built by nich

Made with ⚡ for the AI community

Author

Built by nich - https://x.com/nichxbt

🌐 Live HTTP Deployment

Extract LLMs Docs is deployed and accessible over HTTP via MCP Streamable HTTP transport — no local installation required.

Endpoint:

https://modelcontextprotocol.name/mcp/extract-llms-docs

Connect from any MCP Client

Add to your MCP client configuration (Claude Desktop, Cursor, SperaxOS, etc.):

{
  "mcpServers": {
    "extract-llms-docs": {
      "type": "http",
      "url": "https://modelcontextprotocol.name/mcp/extract-llms-docs"
    }
  }
}

Available Tools (5)

Tool	Description
`fetch_llms_txt`	Fetch llms.txt from a domain
`fetch_llms_full_txt`	Fetch full llms.txt
`extract_documentation`	Extract documentation from URL
`verify_llms_txt`	Verify llms.txt support
`discover_documentation_urls`	Discover doc URLs

Example Requests

Fetch llms.txt from a domain:

curl -X POST https://modelcontextprotocol.name/mcp/extract-llms-docs \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"fetch_llms_txt","arguments":{"url":"https://docs.anthropic.com"}}}'

Fetch full llms.txt:

curl -X POST https://modelcontextprotocol.name/mcp/extract-llms-docs \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"fetch_llms_full_txt","arguments":{"url":"https://docs.anthropic.com"}}}'

Extract documentation from URL:

curl -X POST https://modelcontextprotocol.name/mcp/extract-llms-docs \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"extract_documentation","arguments":{"url":"https://docs.anthropic.com/en/docs","max_length":5000}}}'

List All Tools

curl -X POST https://modelcontextprotocol.name/mcp/extract-llms-docs \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'

Also Available On

SperaxOS — Browse and install from the MCP marketplace
All 27 MCP servers — See the full catalog at modelcontextprotocol.name

Powered by modelcontextprotocol.name — the open MCP HTTP gateway

Star History

Documentation

Full documentation site: https://nirholas.github.io/extract-llms-docs/

Getting started covers install and first run.
Examples has copy-paste snippets.

View this README on GitHub

Установка

npx -y @llm-energy/mcp-server

Конфигурация

{
  "mcpServers": {
    "llm-energy": {
      "command": "npx",
      "args": ["-y", "@llm-energy/mcp-server"]
    }
  }
}

nirholas/extract-llms-docs

Обзор

README

llm.energy

📖 Overview

✨ Key Features

🎯 Use Cases

🚀 install.md Generator

How It Works

Quick Example

📚 Documentation

🚀 Installation

Web Application

Local Development

MCP Server

🔌 API Reference

🤖 MCP Server Tools

🏗️ Architecture

🛠️ Technology Stack

⚙️ Configuration

💻 Development

🔗 Related Projects

🌐 Sites with llms.txt Support

🤝 Contributing

📄 License

🔗 Links

👤 Author

Author

🌐 Live HTTP Deployment

Connect from any MCP Client

Available Tools (5)

Example Requests

List All Tools

Also Available On

Star History

Documentation

Установка

Конфигурация