OWASP Juice Shop Automation with MCP

A Python-based UI and API automation project for OWASP Juice Shop, extended with a Model Context Protocol (MCP) server so an MCP client can discover and invoke QA automation tools.

The project combines Pytest, Selenium WebDriver, Requests, Docker, and MCP. It also experiments with a local Ollama model (qwen2.5:1.5b) as part of the client workflow.

Project status: The full Pytest suite was verified with 3 passing tests. The MCP tools for running tests, retrieving results, project information, screenshots, and failure evidence have been exercised. The current client uses keyword-based request routing, so it should not be described as a fully autonomous AI agent.

Features

• Selenium UI automation for login and payment flows.
• REST API automation with Python Requests.
• Pytest fixtures for shared browser setup and login.
• JSON files for test data.
• Docker-based local Juice Shop environment.
• MCP tools for test execution, result retrieval, screenshots, and failure evidence.
• Structured test summary alongside raw Pytest output.
• Automatic screenshot and traceback text report when a Selenium UI test fails.
• Local Ollama/Qwen experimentation.

Technology Stack

Project Structure

juice-shop-automation-mcp/
├── mcp/
│   └── server.py
├── screenshots/
│   ├── failures/
│   │   ├── _.png
│   │   └── _.txt
│   ├── all-tests.png
│   ├── api-test.png
│   ├── docker-running.png
│   ├── ui-test.png
│   ├── juice-shop-payment.png
│   └── mcp-received-screenshot.png
├── test_data/
│   ├── card-data.json
│   └── new-user.json
├── tests/
│   ├── test_cards_api.py
│   ├── test_login.py
│   └── test_payment.py
├── ai_client.py
├── conftest.py
├── requirements.txt
├── .gitignore
└── README.md

The exact screenshot files in a local checkout may differ depending on which tests have been run.

Architecture

User request
    |
    v
ai_client.py
    |
    | maps request to a tool
    v
MCP client (ClientSession)
    |
    | session.call_tool(...)
    v
mcp/server.py
    |
    +--> get_project_info
    |
    +--> run_tests ------> subprocess ------> Pytest
    |                                         |
    |                                         +--> Selenium UI tests
    |                                         +--> Requests API tests
    |
    +--> get_test_results
    |
    +--> get_screenshot
    |
    +--> get_failure_evidence
                    |
                    v
          Text report + screenshot

MCP provides a standard interface for discovering and calling the project’s QA tools. MCP does not replace Pytest or Selenium: the server invokes Pytest, and the tests use Selenium or Requests to interact with the application.

Current client behavior

The client discovers the tools exposed by the server with session.list_tools(). In the current debugging-friendly version, keyword-based routing maps requests such as run tests, latest results, screenshot, and failure evidence to specific tools. Ollama/Qwen is part of the project experimentation, but this routing path should not be represented as model-driven tool selection or a fully autonomous agent.

MCP Tools

Test-result behavior

The results report includes a status, counts parsed from Pytest’s summary line, duration when available, exit code, and raw Pytest output. Failure output preserves the traceback produced by Pytest.

The current implementation stores results in memory in the MCP server process. They are not persistent across server restarts. The summary parser is intentionally lightweight; unusual interruptions or collection errors may not yield complete test counts.

Test Coverage

The current test modules are:

• tests/test_login.py — UI login behavior.
• tests/test_payment.py — UI payment-card flow.
• tests/test_cards_api.py — card/payment-related API behavior.

The suite was verified with:

3 passed

The number of tests may change as coverage grows. A passing test suite demonstrates that the tested scenarios passed in that run; it does not prove that the entire application is defect-free.

Prerequisites

• Python installed.
• Docker Desktop installed and running.
• Google Chrome installed for UI tests.
• Juice Shop available at http://localhost:3000.
• Dependencies installed from requirements.txt.
• Ollama and the qwen2.5:1.5b model are needed only for the local-model workflow, not for running Pytest directly.

Setup

1. Clone the repository

git clone https://github.com/RVRajkumar/juice-shop-automation-mcp.git
cd juice-shop-automation-mcp

2. Create and activate a virtual environment

Windows PowerShell:

python -m venv .venv
.\.venv\Scripts\Activate.ps1

macOS/Linux:

python -m venv .venv
source .venv/bin/activate

3. Install dependencies

python -m pip install -r requirements.txt

4. Start OWASP Juice Shop

If the image is not already available:

docker pull bkimminich/juice-shop

Start the application:

docker run -d --name juice-shop -p 3000:3000 bkimminich/juice-shop

If a container already exists, check it with docker ps and use docker start juice-shop if it is stopped. If port 3000 is already occupied, resolve the existing container/port conflict rather than repeatedly creating containers.

Open http://localhost:3000 to verify that the application loads.

5. Configure test data and Chrome

• Check the JSON files under test_data/ and provide suitable local test credentials for your Juice Shop setup.
• The current conftest.py specifies a Windows Chrome executable path. If Chrome is installed elsewhere, update options.binary_location to the actual path, or adapt the fixture for your environment.
• Do not commit real passwords, tokens, or other secrets. Use safe test-only credentials and keep sensitive local values out of version control.

Running Tests

Ensure the virtual environment is active and Juice Shop is running.

Run the full suite:

python -m pytest -q

Run with detailed output:

python -m pytest -v

Run the UI payment test and show captured output:

python -m pytest tests/test_payment.py -q -s

Run an individual module:

python -m pytest tests/test_login.py -v
python -m pytest tests/test_payment.py -v
python -m pytest tests/test_cards_api.py -v

Running the MCP Client

Start the client from the project root:

python ai_client.py

The client lists the tools exposed by the server and accepts a request. Supported keyword requests in the current implementation include:

• project info
• run tests
• latest results
• screenshot
• failure evidence
• exit

The current client launches the MCP server as a subprocess. Keep the project dependencies installed in the active .venv and start the client from the repository root.

Failure Evidence Collection

A Pytest hook in conftest.py checks failed setup and test-call phases. For a UI test where the Selenium driver fixture is available, it attempts to save:

• A PNG screenshot of the browser at failure time.
• A TXT report containing the test node ID, failed phase, and Pytest traceback.

Files are written to screenshots/failures/. The MCP tool get_failure_evidence returns the latest report and the corresponding PNG when one exists.

This evidence collection is best-effort: it depends on a live Selenium driver being available. API-only failures do not have a browser screenshot, and failures during fixture setup before a driver is available may not produce one.

Challenges Faced and Resolutions

These are the actual development and debugging issues encountered while building this project.

1. MCP server used the wrong Python environment

Symptom: Test execution through MCP did not initially behave like direct execution from the project environment.

Cause: The server/client was being launched with a Python interpreter from another project environment.

Resolution: Used sys.executable to run Pytest with the interpreter running the MCP server, and verified the printed interpreter and project directory. This made MCP test execution use the V2 project’s .venv.

2. Selenium could not locate the Chrome browser correctly

Symptom: Chrome startup failed through the MCP-launched test process with a Selenium Manager browser-path error.

Resolution: Configured Selenium’s Chrome options with the actual Chrome executable path in conftest.py, then verified that direct Pytest execution and MCP-triggered execution both worked.

Portability note: The configured path is machine-specific and may need to be changed on another computer.

3. get_test_results initially returned “No tests have been run yet”

Symptom: Calling the results tool before running tests returned the initial placeholder.

Cause: The tool reads in-memory results from the current MCP server process; it does not execute tests itself. A new server process starts with no stored run.

Resolution: Tested run_tests and then get_test_results within the same client/server session, and improved the stored result to include a structured summary and the raw Pytest output.

4. MCP screenshot output showed an old or unexpected image

Symptom: The screenshot tool returned an IDE screenshot instead of the desired Juice Shop payment page.

Cause: The tool was reading a different screenshot file from the one intended for this check.

Resolution: Updated the screenshot path to screenshots/juice-shop-payment.png, then verified that the MCP client received and saved the returned PNG as screenshots/mcp-received-screenshot.png.

5. Screenshot content attribute mismatch

Symptom: The client had an issue reading the image MIME type.

Cause: The MCP SDK’s returned image content exposes the Python attribute mime_type, not mimeType.

Resolution: Updated the client to use the SDK’s item.mime_type attribute and verified that the PNG was saved successfully.

6. Failure evidence was not reachable from the client request

Symptom: The MCP server listed get_failure_evidence, but entering failure evidence in the client produced the unsupported-request message.

Cause: The server exposed the new tool, but the client’s keyword-routing logic did not yet map that phrase to the tool name.

Resolution: Added a failure evidence condition to the client routing logic and verified that it returned the saved text report and screenshot.

7. Investigating a real Selenium click-interception failure

Observed behavior: A payment test failed with ElementClickInterceptedException while clicking the card-number field. Selenium reported that another element in cdk-overlay-1 would receive the click.

Status: Investigated using the saved traceback and screenshot, but deliberately left the underlying test issue unchanged for later investigation. This is a recorded observation, not a resolved defect.

8. Verifying failure-reporting behavior

Approach: Temporarily introduced an intentional assertion failure to verify the reporting path.

Result: Confirmed that Pytest failure evidence included a text traceback report and screenshot, then removed the temporary assertion and reran the full suite. The final verified suite result was 3 passed.

Screenshots

Useful screenshots are stored in the screenshots/ directory, including UI/API/Docker evidence, the payment page, and failure evidence. Open the files locally to review them. If you choose to display images directly in GitHub, use repository-relative Markdown image links only for images that are committed to the repository.

Troubleshooting

Juice Shop does not load

Check Docker Desktop and the container:

docker ps

Open http://localhost:3000. Check container logs if it is running but the application is unavailable.

Pytest is missing

Activate the project’s virtual environment and install the dependencies:

.\.venv\Scripts\Activate.ps1
python -m pip install -r requirements.txt
python -m pytest --version

Chrome does not start

Confirm Chrome is installed and that options.binary_location in conftest.py points to the real executable on this machine. Verify the active interpreter with:

python -c "import sys; print(sys.executable)"

MCP does not show a tool

Check that the tool is registered in mcp/server.py, that the server starts without an exception, and that the client routing logic recognizes the requested phrase. The server’s tool list and the client’s request routing are separate parts of the workflow.

Failure screenshot is missing

The hook needs a Selenium driver to be available at the failed phase. API-only failures and setup failures that happen before the driver fixture is ready may produce a text report without a screenshot, or no driver evidence at all.

Limitations and Future Improvements

• Replace keyword-only request routing with verified model-driven tool selection and controlled tool-call handling.
• Add persistent storage for test results across MCP server restarts.
• Improve result parsing for interrupted runs, collection errors, and timeouts.
• Capture structured evidence for API failures as well as UI failures.
• Improve screenshot handling for failures that occur during fixture setup or teardown.
• Make Chrome configuration portable across operating systems.
• Add richer reports, logging, and CI integration.
• Investigate and resolve the observed payment-field click interception separately.

Repository

GitHub repository
GitHub profile

Author

Rajkumar R V