SOFIA - Sort of Functional Interactive Agent

A sophisticated AI assistant platform that combines desktop automation, email/calendar management, and multimodal interaction capabilities. SOFIA provides a flexible, privacy-conscious solution for automating tasks through natural language conversation.

Overview

SOFIA operates through multiple interfaces (web, desktop overlay, and MCP server) with a dual AI backend system supporting both local (Ollama) and cloud (OpenAI) models. The platform uses advanced computer vision for UI automation and integrates seamlessly with Google services while maintaining strict security boundaries.

Key Features

• Desktop Automation: Computer vision-powered UI element detection using OmniParser, precise mouse/keyboard control, screenshot analysis, and sandboxed shell command execution
• Email & Calendar Integration: Full Gmail functionality (search, compose, reply, forward) and Google Calendar management through natural language with OAuth2 authentication
• Local Conversation Storage: Compacts and stores past conversations as md files for future use and reference.
• Multimodal Processing: Text and image chat support, audio file transcription via Whisper, real-time streaming responses, and drag-and-drop file handling
• Flexible AI Backend: Hot-swappable between local Ollama models (privacy-focused, offline) and OpenAI API (faster, more accurate) via Brain Factory pattern
• Safety-First Design: All file operations sandboxed to ~/SOFIA/ directory, parameter validation, graceful error handling, and secure API key management

Architecture

SOFIA employs a modular architecture with three primary interfaces:

1. Web Interface (sofia_web.py): Gradio-based UI for multimodal chat and audio transcription
2. Desktop Interface (sofia_desktop.py): PyQt6 transparent overlay for seamless desktop integration
3. MCP Server (sofia_gmail.py): Model Context Protocol server for Google services

The system uses a THINK → PLAN → EXECUTE → VERIFY workflow for autonomous task completion, with concurrent tool execution and conversation memory management.

Demo

See SOFIA’s computer use capabilities in action:

sofia_demo.webm

The demo showcases SOFIA’s ability to understand visual interfaces, navigate applications, and perform desktop automation tasks.

Getting Started

Requirements

• Ubuntu 22.04+ (Windows/macOS support planned)
• Python 3.11+
• For computer use and local AI: CUDA enabled GPU (24gb+ recommended)
• For cloud AI: OpenAI API key
• Google credentials for email/calendar features
• Separate monitor recommended for desktop automation

Installation

git clone https://github.com/akim42003/SOFIA.git
cd SOFIA
conda create -n sofia python=3.11
conda activate sofia
pip install -r requirements.txt

# For local AI models
ollama create sofia -f Modelfile.enhanced