Marketplace | Documentation | Issues

Skill Eval Action

A GitHub Action that evaluates Claude Code skills against YAML test cases with automated grading and PR reporting.

Usage

Single skill

- uses: skill-bench/skill-eval-action@v1
  with:
    skill-name: tf-guide
    skill-path: ./skills/tf-guide
    anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }}

Multiple skills (static matrix)

Run skills in parallel - each skill gets its own job:

name: Skill Eval
on:
  pull_request:
    paths:
      - 'skills/**'

permissions:
  contents: read
  pull-requests: write

jobs:
  eval:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    strategy:
      fail-fast: false
      matrix:
        skill:
          - tf-guide
          - k8s-operator-sdk
          - secure-gh-workflow
    steps:
      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6

      - uses: skill-bench/skill-eval-action@v1
        with:
          skill-name: ${{ matrix.skill }}
          skill-path: skills/${{ matrix.skill }}
          anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }}
          pass-threshold: '80'

Auto-discover all skills (dynamic matrix)

Automatically find and evaluate all skills that have evals/ directories - no need to hardcode skill names:

name: Skill Eval
on:
  pull_request:
    paths:
      - 'skills/**'

permissions:
  contents: read
  pull-requests: write

jobs:
  discover:
    runs-on: ubuntu-latest
    outputs:
      skills: ${{ steps.discover.outputs.skills }}
      count: ${{ steps.discover.outputs.count }}
    steps:
      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
        with:
          persist-credentials: false
          sparse-checkout: skills

      - name: Discover skills with evals
        id: discover
        run: |
          skills=$(find skills -name "*.yaml" -path "*/evals/*" -exec dirname {} \; | xargs -I{} dirname {} | xargs -I{} basename {} | sort -u | jq -R -s -c 'split("\n") | map(select(. != ""))')
          echo "skills=$skills" >> "$GITHUB_OUTPUT"
          echo "count=$(echo $skills | jq length)" >> "$GITHUB_OUTPUT"

      - name: Summary
        run: echo "Found ${{ steps.discover.outputs.count }} skills with evals"

  eval:
    needs: discover
    if: needs.discover.outputs.count > 0
    runs-on: ubuntu-latest
    timeout-minutes: 30
    strategy: