一个复杂的人工智能助手平台，结合了桌面自动化、电子邮件/日历管理和多模态交互功能。SOFIA为通过自然语言对话自动化任务提供了灵活且注重隐私的解决方案。

概述

SOFIA通过多种接口（网页、桌面叠加和MCP服务器）运行，配备双AI后端系统，支持本地（Ollama）和云端（OpenAI）模型。该平台采用先进的计算机视觉实现UI自动化，并能无缝集成Google服务，同时保持严格的安全边界。

主要特色

• 桌面自动化：利用OmniParser实现计算机视觉驱动的UI元素检测，精确的鼠标/键盘控制、截图分析和沙盒壳命令执行
• 电子邮件与日历集成：完整的Gmail功能（搜索、写作、回复、转发）及通过自然语言和OAuth2认证的Google日历管理
• 本地会话存储：将过去的对话压缩并存储为 md 文件，供未来使用和参考。

SOFIA - Sort of Functional Interactive Agent

A sophisticated AI assistant platform that combines desktop automation, email/calendar management, and multimodal interaction capabilities. SOFIA provides a flexible, privacy-conscious solution for automating tasks through natural language conversation.

Overview

SOFIA operates through multiple interfaces (web, desktop overlay, and MCP server) with a dual AI backend system supporting both local (Ollama) and cloud (OpenAI) models. The platform uses advanced computer vision for UI automation and integrates seamlessly with Google services while maintaining strict security boundaries.

Key Features

• Desktop Automation: Computer vision-powered UI element detection using OmniParser, precise mouse/keyboard control, screenshot analysis, and sandboxed shell command execution
• Email & Calendar Integration: Full Gmail functionality (search, compose, reply, forward) and Google Calendar management through natural language with OAuth2 authentication
• Local Conversation Storage: Compacts and stores past conversations as md files for future use and reference.
• Multimodal Processing: Text and image chat support, audio file transcription via Whisper, real-time streaming responses, and drag-and-drop file handling
• Flexible AI Backend: Hot-swappable between local Ollama models (privacy-focused, offline) and OpenAI API (faster, more accurate) via Brain Factory pattern
• Safety-First Design: All file operations sandboxed to ~/SOFIA/ directory, parameter validation, graceful error handling, and secure API key management

Architecture

SOFIA employs a modular architecture with three primary interfaces:

1. Web Interface (sofia_web.py): Gradio-based UI for multimodal chat and audio transcription
2. Desktop Interface (sofia_desktop.py): PyQt6 transparent overlay for seamless desktop integration
3. MCP Server (sofia_gmail.py): Model Context Protocol server for Google services

The system uses a THINK → PLAN → EXECUTE → VERIFY workflow for autonomous task completion, with concurrent tool execution and conversation memory management.

Demo

See SOFIA’s computer use capabilities in action:

sofia_demo.webm

The demo showcases SOFIA’s ability to understand visual interfaces, navigate applications, and perform desktop automation tasks.

Getting Started

Requirements

• Ubuntu 22.04+ (Windows/macOS support planned)
• Python 3.11+
• For computer use and local AI: CUDA enabled GPU (24gb+ recommended)
• For cloud AI: OpenAI API key
• Google credentials for email/calendar features
• Separate monitor recommended for desktop automation

Installation

git clone https://github.com/akim42003/SOFIA.git
cd SOFIA
conda create -n sofia python=3.11
conda activate sofia
pip install -r requirements.txt

# For local AI models
ollama create sofia -f Modelfile.enhanced