Model Context Protocol (MCP) 서버의 종합 평가 라이브러리. 기본 메트릭, 보안 검증 및 옵션 LLM 기반 평가를 포함한 완전한 MCP 사양 적용으로 MCP 서버를 테스트하고 검증합니다.

Status: MVP – APIstable, 1.0.0 이전에 가능한 최소의 끊기 변화

---

0. 빠른 시작

##1. 설치

##2. 설정파일 만들기

config 파일 생성 (예, mcp-eval.config.ts):

##3. 실행 평가

---

# 1 핵심 개념

MCPVals는 모든 MCP 명세 primitives를 위한 포괄적인 테스트를 제공합니다:

1. Tool Health Testing: 특정 인수로 개별 도구를 직접 호출하여 정확한, 성능 및 오류 처리 확인. 이것은 단위 테스트 및 회귀 검사에 이상적입니다.

2. ** Workflow Evaluation**: 자연적인 언어 프롬프트를 해석하기 위하여 큰 언어 모형 (LLM)를 이용하고 목표 달성하기 위하여 공구 통화의 시리즈를 실행하십시오. 이 테스트는 MCP primitives의 LLM의 관점에서.

---

# 2. 설치 및 실행 요구 사항

1. **Node.js ≥ 18 ** - 우리는 기본적으로 의존 fetch, EventSource], 및 fs/promises].
2. 명세.

MCPVals

A comprehensive evaluation library for Model Context Protocol (MCP) servers. Test and validate your MCP servers with complete MCP specification coverage including Tools with deterministic metrics, security validation, and optional LLM-based evaluation.

Status: MVP – API stable, minor breaking changes possible before 1.0.0

---

0. Quick Start

1. Installation

# Install – pick your favourite package manager
pnpm add -D mcpvals            # dev-dependency is typical

2. Create a config file

Create a config file (e.g., mcp-eval.config.ts):

import type { Config } from "mcpvals";

export default {
  server: {
    transport: "stdio",
    command: "node",
    args: ["./example/simple-mcp-server.js"],
  },

  // Test individual tools directly
  toolHealthSuites: [
    {
      name: "Calculator Health Tests",
      tests: [
        {
          name: "add",
          args: { a: 5, b: 3 },
          expectedResult: 8,
          maxLatency: 500,
        },
        {
          name: "divide",
          args: { a: 10, b: 0 },
          expectedError: "division by zero",
        },
      ],
    },
  ],

  // Test multi-step, LLM-driven workflows
  workflows: [
    {
      name: "Multi-step Calculation",
      steps: [
        {
          user: "Calculate (5 + 3) * 2, then divide by 4",
          expectedState: "4",
        },
      ],
      expectTools: ["add", "multiply", "divide"],
    },
  ],

  // Optional LLM judge
  llmJudge: true,
  openaiKey: process.env.OPENAI_API_KEY,
  passThreshold: 0.8,
} satisfies Config;

3. Run Evaluation

# Required for workflow execution
export ANTHROPIC_API_KEY="sk-ant-..."

# Optional for LLM judge
export OPENAI_API_KEY="sk-..."

# Run everything
npx mcpvals eval mcp-eval.config.ts

# Run only tool health tests
npx mcpvals eval mcp-eval.config.ts --tool-health-only

# Run with LLM judge and save report
npx mcpvals eval mcp-eval.config.ts --llm-judge --reporter json > report.json

---

1. Core Concepts

MCPVals provides comprehensive testing for all MCP specification primitives:

1. Tool Health Testing: Directly calls individual tools with specific arguments to verify their correctness, performance, and error handling. This is ideal for unit testing and regression checking.

2. Workflow Evaluation: Uses a large language model (LLM) to interpret natural language prompts and execute a series of tool calls to achieve a goal. This tests the integration of your MCP primitives from an LLM’s perspective.

---

2. Installation & Runtime Requirements

1. Node.js ≥ 18 – we rely on native fetch, EventSource, and fs/promises.
2. pnpm / npm / yarn – whichever you prefer, MCPVals is published as an ESM‐only package.
3. MCP Server – a local stdio binary or a remote Streaming-HTTP endpoint.