rayss868/mcp-web-curl

本页由 AI 翻译，技术细节请以原文为准。查看原文

11 stars 3 forks 质量 81 趋势 22MIT

Lightweight MCP server and CLI that lets AI fetch and process web content or APIs with Puppeteer, smart commands, search tools, and reliable downloads.

概览

🚀 🛠️ Built with Node.js & TypeScript (Node.js v18+ required) Click here to watch the demo video directly in your browser. If your platform supports it, you can also download and play demo/demo.mp4 directly. Your browser does not support the video tag. See CHANGELOG.md for a complete history of updates and new features. is a powerful tool for fetching and extracting text content from web pages and APIs. Use it as a standalone CLI or as an MCP (Model Context Protocol) server. Web-curl leverages Puppeteer for robust web scraping and supports advanced features such as resource blocking, custom headers, authentication, and External Search API integration. - : Full control over Chromium via Puppeteer (click, type, scroll, hover, key presses). - : Browser profiles are now always persistent. Login sessions, cookies, and cache are automatically saved in a local user_data/ directory.

README

以下为原文 README 节选，完整内容请查看原文。

Web-curl

Developed by Rayss

🚀 Open Source Project
🛠️ Built with Node.js & TypeScript (Node.js v18+ required)

🎬 Demo Video

Click here to watch the demo video directly in your browser.

If your platform supports it, you can also download and play demo/demo.mp4 directly.

📝 Changelog / Update History

See CHANGELOG.md for a complete history of updates and new features.

📝 Overview

Web-curl is a powerful tool for fetching and extracting text content from web pages and APIs. Use it as a standalone CLI or as an MCP (Model Context Protocol) server. Web-curl leverages Puppeteer for robust web scraping and supports advanced features such as resource blocking, custom headers, authentication, and External Search API integration.

✨ Features

🚀 Deep Research & Automation (v1.4.2)

Advanced Browser Automation: Full control over Chromium via Puppeteer (click, type, scroll, hover, key presses).
Always-On Session Persistence: Browser profiles are now always persistent. Login sessions, cookies, and cache are automatically saved in a local user_data/ directory.
Token-Efficient Snapshots (available via the hidden browser_snapshot handler):
- Accessibility Tree: Clean, structured snapshots instead of messy HTML.
- HTML Slice Mode: Raw HTML with startIndex/endIndex for safe chunking when needed.

在 GitHub 查看完整 README

安装

This server does not publish a one-line install command.

Open the repository installation guide 新手？先看 mcpskill.org 教程 →

配置

{
  "mcpServers": {
    "web-curl": {
      "command": "node",
      "args": [
        "build/index.js"
      ],
      "disabled": false,
      "alwaysAllow": [
        "extract",
        "crawl",
        "agent",
        "research",
        "browser_configure",
        "browser_close",
        "multi_search",
        "fetch_api",
        "download_file",
        "parse_document"
      ],
      "env": {
        "SEARCH_BASE_URL": "https://example.com/v1/search",
        "SEARCH_MODEL": "search-combo",
        "SEARCH_API_KEY": "YOUR_EXTERNAL_SEARCH_API_KEY",
        "APIKEY_GOOGLE_SEARCH": "YOUR_GOOGLE_API_KEY",
        "CX_GOOGLE_SEARCH": "YOUR_CX_ID"
      }
    }
  }
}