OmniGet MCP Server (Headless Rust Edition)

Universal Social Media, Web & Document Extraction Engine for AI Agents
Originally created by tonhowtf | Re-architected, enhanced & maintained as a standalone headless Rust edition by Greedy Gains for non-commercial community benefit.

---

💡 Overview

OmniGet MCP Server is an ultra-fast, headless Model Context Protocol (MCP) server and REST API built in Native Rust. It equips AI Agents (such as Claude Desktop, Cursor, and ChatGPT) with the ability to extract, parse, and ingest rich content from social networks, modern web articles, and documents in real time.

Unlike traditional scrapers that rely on heavy browser automation (Chromium / Puppeteer / Playwright), OmniGet MCP operates 100% headlessly using asynchronous Rust HTTP engines and embedded state decoders. It responds in 1–2 seconds, consumes less than 30 MB of RAM, and runs seamlessly in minimal container environments.

---

✨ Key Features

• 🚀 100% Headless Native Rust: Zero browser overhead. No Chromium, No WebKit, and No X11 libraries required.
• 📱 Facebook Post & Media Extraction (Breakthrough):
  • Extracts 100% complete, untruncated captions without text truncation (...).
  • Decodes embedded Relay/GraphQL state and Bloks data structures directly from raw HTML.
  • Automatically resolves share links and slugged URLs into canonical formats, bypassing cloud datacenter login walls.
  • Retrieves high-resolution photo galleries, video streams, and hashtags.
• 📸 Instagram Posts, Reels & Carousels:
  • Full caption and hashtag extraction.
  • Multi-image carousel parsing with direct CDN asset URLs.
• 🐦 X / Twitter Thread Unrolling:
  • Extracts single posts (x_post) with media and metrics.
  • Unrolls full multi-tweet threads (x_thread) into continuous, structured reading material.
• 🌐 Universal Web-to-Markdown:
  • Strips ads, navigation, scripts, and clutter from any public article.
  • Converts web pages into clean, token-efficient Markdown optimized for LLMs.
• 📄 PDF Document Extraction:
  • Ingests and extracts clean plain-text from remote URLs or local PDF files.
• 🎬 Universal Media Info:
  • Powered by yt-dlp integration supporting metadata extraction for 1,800+ media sites.
• 🔒 Enterprise-Grade Security:
  • Timing-safe Bearer token authentication (AUTH_TOKEN) using constant-time byte comparison (constant_time_eq) to protect all operational endpoints.
• ⚡ Dual Protocol Support:
  • MCP over SSE (GET /sse + POST /messages)
  • Streamable HTTP MCP (POST /mcp JSON-RPC 2.0)
  • REST API + OpenAPI 3.1 (GET /openapi.json + POST /api/*)

---

🛠 Available MCP Tools

When connected to an MCP client (Claude Desktop, Cursor, etc.), the server exposes 7 powerful extraction tools:

---

🚀 Quickstart

1. Run with Docker (Recommended)

# Build the Docker image
docker build -t omniget-mcp-server .

# Run the container
docker run -d \
  -p 8080:8080 \
  -e AUTH_TOKEN="your-secure-token" \
  --name omniget-mcp \
  omniget-mcp-server

Check health status:

curl http://localhost:8080/health
# Response: {"ok": true}

---

2. Run Locally with Cargo

Ensure you have Rust 1.80+ installed:

# Clone the repository
git clone https://github.com/greedygains/omniget-mcp-server.git
cd omniget-mcp-server

# Set your authentication token and run
export AUTH_TOKEN="your-secure-token"
cargo run -p omniget-server --release

The server will listen on http://0.0.0.0:8080.

---

3. Deploy to Railway

1. Fork or push this repository to your GitHub account.
2. Create a New Project in Railway and select Deploy from GitHub repo.
3. Under Variables, add:
   • AUTH_TOKEN: A secure random token (e.g. openssl rand -hex 16).
4. Railway will automatically build the Dockerfile, assign an HTTPS domain, and manage the $PORT variable.

---

🔌 Connecting to AI Clients

Claude Desktop

Edit your claude_desktop_config.json:

• macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
• Windows: %APPDATA%\Claude\claude_desktop_config.json

{
  "mcpServers": {
    "omniget": {
      "url": "https://.up.railway.app/sse?token="
    }
  }
}

Or using Bearer header syntax:

{
  "mcpServers": {
    "omniget": {
      "url": "https://.up.railway.app/sse",
      "headers": {
        "Authorization": "Bearer "
      }
    }
  }
}

---

Cursor IDE

Add to ~/.cursor/mcp.json or project-level .cursor/mcp.json:

{
  "mcpServers": {
    "omniget": {
      "url": "https://.up.railway.app/sse?token="
    }
  }
}

---

ChatGPT (Custom Actions)

1. In ChatGPT GPT Builder, go to Configure > Actions > Create new action.
2. Set Authentication to Bearer and input your ``.
3. Import the OpenAPI schema from: https://.up.railway.app/openapi.json?token=

---

📡 API Endpoints

---

🤝 Acknowledgments & Attribution

This open-source project is built on the foundations of the open-source community:

• Upstream Project: The extraction core was originally derived from OmniGet by tonhowtf, a multi-platform downloader application. We express our immense gratitude for their pioneering work on universal media extraction logic.
• Greedy Gains Enhancements: Greedy Gains enhanced and maintains this headless edition for non-commercial community benefit:
  • Decoupled the engine into a standalone, headless microservice free of desktop/Tauri dependencies.
  • Implemented the full Model Context Protocol (MCP) specification over Server-Sent Events (SSE) and Streamable HTTP.
  • Architected the Native Rust Facebook & Instagram extraction engine, enabling full 100% untruncated caption recovery and cloud datacenter login wall bypass.
  • Engineered timing-safe authentication, containerization, and the REST/OpenAPI 3.1 bridge.

---

📄 License

This project is licensed under the GNU General Public License v3.0 (GPL-3.0) in accordance with the upstream OmniGet project.
See the LICENSE file for complete details.