一个用于网络内容扫描和分析的"模型背景协议"(MCP)服务器. 该服务器提供了从网页获取、分析和提取信息的工具。

□ 特性

• 页取取:将网页转换为Markdown,以便于分析
• ** Link 提取器**: 从网页提取和分析链接
• Site Crawling:反复地爬行网站以发现内容
• 链接检查: 在网页上识别断开的链接
• ** 模式匹配**:查找匹配特定模式的 URL
• ** 物品生成**: 生成网站的 XML 站点地图

□ 安装

通过 Smithery 安装

要通过 Smithery 自动为 Claude 桌面安装 Webscan :

手动安装

□ 使用量

QQ 启动服务器

服务器运行在stdio传输上,使其与克洛德桌面等MCP客户端兼容.

可用的工具

1. fetch-page

• 获取网页并将其转换为Markdown。
• 参数:
• url(需要):要获取的页面的URL。
• selector(可选):CSS选择器以瞄准特定内容.

2. extract-links

• 从网页上提取所有链接及其文本。
• 参数:
• url(需要):页面的URL进行分析.

MCP Webscan Server

A Model Context Protocol (MCP) server for web content scanning and analysis. This server provides tools for fetching, analyzing, and extracting information from web pages.

Features

• Page Fetching: Convert web pages to Markdown for easy analysis
• Link Extraction: Extract and analyze links from web pages
• Site Crawling: Recursively crawl websites to discover content
• Link Checking: Identify broken links on web pages
• Pattern Matching: Find URLs matching specific patterns
• Sitemap Generation: Generate XML sitemaps for websites

Installation

Installing via Smithery

To install Webscan for Claude Desktop automatically via Smithery:

npx -y @smithery/cli install mcp-server-webscan --client claude

Manual Installation

# Clone the repository
git clone 
cd mcp-server-webscan

# Install dependencies
npm install

# Build the project
npm run build

Usage

Starting the Server

npm start

The server runs on stdio transport, making it compatible with MCP clients like Claude Desktop.

Available Tools

1. fetch-page

   • Fetches a web page and converts it to Markdown.
   • Parameters:
     • url (required): URL of the page to fetch.
     • selector (optional): CSS selector to target specific content.

2. extract-links

   • Extracts all links from a web page with their text.
   • Parameters:
     • url (required): URL of the page to analyze.
     • baseUrl (optional): Base URL to filter links.
     • limit (optional, default: 100): Maximum number of links to return.

3. crawl-site

   • Recursively crawls a website up to a specified depth.
   • Parameters:
     • url (required): Starting URL to crawl.
     • maxDepth (optional, default: 2): Maximum crawl depth (0-5).

4. check-links

   • Checks for broken links on a page.
   • Parameters:
     • url (required): URL to check links for.

5. find-patterns

   • Finds URLs matching a specific pattern.
   • Parameters:
     • url (required): URL to search in.
     • pattern (required): JavaScript-compatible regex pattern to match URLs against.

6. generate-site-map

   • Generates a simple XML sitemap by crawling.
   • Parameters:
     • url (required): Root URL for sitemap crawl.
     • maxDepth (optional, default: 2): Maximum crawl depth for discovering URLs (0-5).
     • limit (optional, default: 1000): Maximum number of URLs to include in the sitemap.