Llama Stack with MCP Server

Welcome to the Llama Stack with MCP Server Kickstart!

Use this to quickly deploy Llama 3.2-3B on vLLM with Llama Stack and MCP servers in your OpenShift AI environment.

To see how it’s done, jump straight to installation.

Table of Contents

1. Description
2. Custom MCP Server
3. How MCP Servers Work with Llama Stack
   • MCP Server Registration
   • Tool Execution Flow
4. Architecture diagrams
5. References
6. Prerequisites
   • Minimum hardware requirements
   • Required software
   • Required permissions
7. Install
   • Clone the repository
   • Create the project
   • Single Command Installation (Recommended)
8. Test
9. Cleanup

Description

This kickstart provides a complete setup for deploying:

• Llama 3.2-3B model using vLLM on OpenShift AI
• Llama Stack for agent-based interactions
• Sample HR application providing restful services to HR data e.g. vacation booking
• MCP Weather Server for real-time weather data access
• Custom MCP server providing access to the sample HR application

Custom MCP Server

The custom MCP server (custom-mcp-server/) demonstrates how to build a Model Context Protocol server that integrates with enterprise APIs. This server provides the following tools to the LLM:

• Vacation Management: Check vacation balances and create vacation requests

The MCP server acts as a bridge between the Llama Stack and the HR Enterprise API, translating LLM tool calls into REST API requests.

Source Code & Build Instructions: If you want to modify the custom MCP server, see the complete source code and build instructions in the custom-mcp-server/ directory. The server is built using Python and can be customized to integrate with your own enterprise APIs.

How MCP Servers Work with Llama Stack

MCP Server Registration

MCP servers are registered with Llama Stack through configuration. The Llama Stack server automatically discovers and connects to configured MCP servers at startup. Here’s an example of how MCP servers are configured:

# Llama Stack MCP server configuration
mcpServers:
  - name: "mcp-weather"
    uri: "http://mcp-weather:3001"
    description: "Weather data MCP server"
  - name: "hr-api-tools"
    uri: "http://custom-mcp-server:8000/sse"
    description: "HR API MCP server with employee, vacation, job, and performance tools"

In this example, this configuration is maintained in the llama-stack-config config map, part of the llama-stack helm chart.

When Llama Stack starts, it: