针对多智能体运行时的可重用 XCrawl 技能定义，重点关注 API 优先的网络数据工作流。 规范仓库：https://github.com/xcrawl-api/xcrawl-skills

英文 |中文

什么是XCrawl

XCrawl 是一款用于搜索、抓取、URL 映射和网站爬取的网络数据基础设施产品。 该仓库提供面向生产环境的技能定义，帮助代理一致调用 XCrawl API。

本仓库所提供的内容

• 适用于常见 XCrawl 工作流程的现成技能
• 带有请求/响应参数文档的面向 API 指令
• 适合运行时执行的cURL和Node示例
• 多智能体编排的统一合同

技能目录

• xcrawl：默认的XCrawl条目技能，用于直接查找和单一URL提取
• xcrawl-scrape：单一 URL 提取和结构化数据工作流程
• xcrawl-map：网站URL发现和范围规划工作流程
• xcrawl-crawl：批量站点爬取和异步结果处理工作流程
• xcrawl-search：基于查询的发现，支持位置/语言控制

快速启动

1.前提条件

• XCrawl API 密钥
• 注册于’https://dash.xcrawl.

XCrawl Skills

Reusable XCrawl skill definitions for multi-agent runtimes, focused on API-first web data workflows. Canonical repository: https://github.com/xcrawl-api/xcrawl-skills

English | 中文

What Is XCrawl

XCrawl is a web data infrastructure product for search, scraping, URL mapping, and site crawling. This repository provides production-oriented skill definitions that help agents call XCrawl APIs consistently.

What This Repository Provides

• Ready-to-use skills for common XCrawl workflows
• API-oriented instructions with request/response parameter documentation
• cURL and Node examples suitable for runtime execution
• A consistent contract for multi-agent orchestration

Skill Catalog

• xcrawl: Default XCrawl entry skill for direct lookup and single-URL extraction
• xcrawl-scrape: Single-URL extraction and structured data workflows
• xcrawl-map: Site URL discovery and scope planning workflows
• xcrawl-crawl: Bulk site crawling and async result handling workflows
• xcrawl-search: Query-based discovery with location/language controls

Quick Start

1. Prerequisites

• An XCrawl API key
• Register at https://dash.xcrawl.com/ and activate the free 1000 credits plan
• Runtime binaries: curl and node
• Access to this repository

2. Configure Local API Key

Create local config file:

Path: ~/.xcrawl/config.json

{
  "XCRAWL_API_KEY": ""
}

Skills in this repo are designed to read XCRAWL_API_KEY from this local file.

3. Choose a Skill

Open one of:

• skills/xcrawl/SKILL.md
• skills/xcrawl-scrape/SKILL.md
• skills/xcrawl-map/SKILL.md
• skills/xcrawl-crawl/SKILL.md
• skills/xcrawl-search/SKILL.md

Each skill includes:

• Applicable scenarios
• Request parameters
• Response parameters
• cURL / Node examples

4. Run Requests

Use the examples in each SKILL.md directly, then adapt request payloads for your business scenario.

Example User Intents

• “Scrape this page in sync mode and return markdown + links.”
• “Map only /docs/ URLs under this domain with a limit of 2000.”
• “Start a bounded crawl (depth 2, limit 100) and poll until completed.”
• “Search in US English and return top 20 results for this query.”

Cross-Agent Contract

Each skill can be executed through a runtime adapter layer.

• Input normalization: goal, inputs, constraints, credentials_ref, runtime_context
• Output normalization: status, request_payload, raw_response or async pair, task_ids, error

Default behavior is raw passthrough: return upstream API response bodies as-is.

Shared XCrawl Conventions

• Base URL: https://run.xcrawl.com
• Local config file: ~/.xcrawl/config.json
• API key field in local config: XCRAWL_API_KEY
• Auth header: Authorization: Bearer
• Main endpoints:
  • POST /v1/scrape and GET /v1/scrape/{scrape_id}
  • POST /v1/map
  • POST /v1/crawl and GET /v1/crawl/{crawl_id}