PySpark 被 LakeSail 淹没的 MPC 服务器
概览
PySpark 磁共振 服务器是用于Apache Spark的"模型背景协议"(MCP)的轻量级服务器执行. 这个MCP服务器的主要目的是利用AI系统促进查询优化. 它提供了从Spark到AI系统的逻辑和物理查询计划以供分析,以及额外的查询计划信息. 此外,服务器还披露目录和表格信息,使Spark提供动力的数据湖具备数据发现能力。 安装后, 请使用 pyspark- mcp 命令启动服务器 : CLI自动处理火花投放配置. 支持所有标准火花投放选项 : 所有火花投放选项(-conf,-jars,-packages,-executor-memory等)都自动通过. - Python = 3.11, = 2.10.6 - loguru - pyspark = 3.5 PySpark MCP 服务器包含以下工具:
概述（AI 翻译）
hi# PySpark MCP 服务器
□ 说明
PySpark 磁共振 服务器是用于Apache Spark的"模型背景协议"(MCP)的轻量级服务器执行.
这个MCP服务器的主要目的是利用AI系统促进查询优化. 它提供了从Spark到AI系统的逻辑和物理查询计划以供分析,以及额外的查询计划信息. 此外,服务器还披露目录和表格信息,使Spark提供动力的数据湖具备数据发现能力。
□ 快开始
- 安装
QQ 运行服务器
安装后,使用 pyspark-mcp 命令启动服务器 :
CLI自动处理spark-submit配置. 支持所有标准火花投放选项 :
QQ CLI 选项
\ 选项\ 默认\ \ 描述\ \
|------------.
QQ [ [T2] [T3] [T3] 火花主URL QQ
QQ --host [T5] QQ MCP服务器主机地址 QQ
| [[T6] [T7] | MCP服务器端口号
--spark-submit[T9] 闪光可执行文件的路径
QQ[ [T10]] - 未执行的打印命令 QQ
所有火花投放选项(--conf),[T12],[T13],[T14]等.
本页由 AI 翻译，技术细节请以原文为准。
README
以下为原文 README 节选，完整内容请查看原文。
hi# PySpark MCP Server
Description
PySpark MCP Server is a lightweight server implementation of Model Context Protocol (MCP) for Apache Spark.
The primary purpose of this MCP server is to facilitate query optimization using AI systems. It provides both logical and physical query plans from Spark to AI systems for analysis, along with additional query plan information. Furthermore, the server exposes catalog and table information, enabling data discovery capabilities in data lakes powered by Spark.
Quick Start
Installation
pip install pyspark-mcp
Running the Server
After installation, use the pyspark-mcp command to start the server:
pyspark-mcp --master "local[*]" --host 127.0.0.1 --port 8090
The CLI automatically handles spark-submit configuration. All standard spark-submit options are supported:
# With additional Spark configuration
pyspark-mcp --master "local[*]" --conf spark.driver.memory=4g
# YARN cluster mode
pyspark-mcp --master yarn --deploy-mode client --num-executors 4
# With additional JARs
pyspark-mcp --master "local[*]" --jars /path/to/connector.jar
# Preview the spark-submit command without running
pyspark-mcp --master "local[*]" --dry-run
# With GraphFrames package
pyspark-mcp --master "local[*]" --packages io.graphframes:graphframes-spark3_2.12:0.10.1
CLI Options
| Option | Default | Description |
|---|---|---|
--master |
local[*] |
Spark master URL |
--host |
127.0.0.1 |
MCP server host address |
--port |
8090 |
MCP server port number |
--spark-submit |
spark-submit |
Path to spark-submit executable |
--dry-run |
- | Print command without executing |
All spark-submit options (--conf, --jars, --packages, --executor-memory, etc.) are passed through automatically.
Adding the running MCP to the Claude-code
# Must run one server on a different port per Claude instance
claude mcp add --transport http pyspark-mcp http://127.0.0.1:8090/mcp
Dependencies
- Python >=3.11,<4.0
- fastmcp >= 2.10.6
- loguru
- pyspark >= 3.5
Bundled MCP tools
The following tools are included in the PySpark MCP Server:
| MCP Tool | Description |
|---|---|
| Get the version of PySpark | Get the version number from the current PySpark Session |
| Get Analyzed Plan of the query | Extracts an analyzed logical plan from the provided SQL query |
| Get Optimized Plan of the query | Extracts an optimized logical plan from the provided SQL query |
安装
This server does not publish a one-line install command.
Open the repository installation guide 新手？先看 mcpskill.org 教程 →