Local Email Agent

An intelligent email assistant powered by Microsoft’s Phi-4 local model, featuring a modern web UI, human-in-the-loop approvals, calendar integration, and Microsoft 365 connectivity via MCP (Model Context Protocol).

Overview

This project demonstrates how to build a local email agent using:

• Foundry Local (runs entirely on your machine), this project shows phi-4 but you can use any model you like
• PostgreSQL + pgvector for semantic email search
• Microsoft 365 MCP Server for email/calendar operations
• LangGraph for agent orchestration with Human-in-the-Loop (HITL) approvals
• FastAPI Backend with SSE streaming for real-time updates
• Next.js Frontend - Agent Inbox UI for managing conversations and approvals

🌱 Quick Start

Prerequisites

1. Python 3.11+
2. Docker (for PostgreSQL)
3. Node.js 18+ (for MCP server and frontend)

Setup

1. Install Foundry Local:

Windows:

winget install Microsoft.FoundryLocal

MacOS:

brew install microsoft/foundrylocal/foundrylocal

Note: Foundry Local doesn’t currently support Linux

2. Install Dependencies:

# Create virtual environment
python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install Python packages
pip install -r email_agent/requirements.txt

# Install Microsoft 365 MCP Server
npm install -g @softeria/ms-365-mcp-server

# Login to the M365 account you want to use
npx @softeria/ms-365-mcp-server --login

# Install frontend dependencies
cd email_agent/frontend
npm install
cd ../..

3. Start PostgreSQL + pgvector:

# Start Docker container
docker compose up -d

# Wait ~10 seconds for PostgreSQL to initialize

# Enable pgvector extension
docker exec -it email-postgres psql -U postgres -d emaildb -c "CREATE EXTENSION IF NOT EXISTS vector;"

# Verify
docker exec -it email-postgres psql -U postgres -d emaildb -c "\dx"

If successful, you should see the vector extension listed:

  Name   | Version |   Schema   |                        Description
---------+---------+------------+-----------------------------------------------------------
 plpgsql | 1.0     | pg_catalog | PL/pgSQL procedural language
 vector  | 0.5.0   | public     | vector data type and ivfflat and hnsw access methods

4. Configure Environment:

# Copy local env template
cp email_agent/.env.local.example email_agent/.env

Edit the .env file and add your Azure OpenAI credentials:

AZURE_OPENAI_ENDPOINT=https://your-endpoint.openai.azure.com/
AZURE_OPENAI_API_KEY=your-api-key
AZURE_OPENAI_EMBEDDING_DEPLOYMENT=text-embedding-ada-002

5. Import Your Emails (Optional)

# Import last 3 months of emails from Outlook
python3 -m email_agent.import_emails --months 3 --batch-size 50 --storage local

6. Run the Agent:

Start both the backend and frontend: