google-ai-edge/litert-samples

---

Welcome to litert-samples — the central developer hub for

• LiteRT (Google’s high-performance, cross-platform, production-proven on-device inference, powering Google and 3P applications globally, succeeding TensorFlow Lite) and
• LiteRT-LM (Google’s specialized orchestration layer for running LLMs with LiteRT, unlocking maximum performance and efficiency).

Whether you want to try a model in your browser in 30 seconds, drop a pre-converted model from Hugging Face litert-community into a mobile/desktop/web app, or convert, quantize, verify, and benchmark a new checkpoint on CPU, GPU, and NPU, this repository takes you from first run to production deployment.

---

🔥 What’s New

• 🦋 Cross-Platform Flutter & Intel NPU Showcases: Added a full multimodal Flutter app (samples/litert/flutter/) running voice chat, live camera detection, RAG, and agent skills across Android, iOS, macOS, Linux, Raspberry Pi 5, Jetson Orin, and Qualcomm VENTUNO Q — plus a 3-act Intel Core Ultra NPU showcase (samples/litert/flutter_intel_npu/) running Gemma 4 across CPU, GPU, and NPU.
• 📱 LiteRT Model Zoo (34 Tasks in One App): Expanded samples/litert_model_zoo/ to 21 Vision, 8 Audio, and 5 Text Decision models via the LiteRT CompiledModel API, streaming verified weights on-demand from litert-community.
• 🎙️ Offline Voice Assistant & Embodied Robotics: Added an offline Android voice assistant with tool calling (samples/litert_lm/voice_assistant/) and a Raspberry Pi Reachy voice robot that sees, hears, reasons, and speaks (samples/litert_lm/reachy-voice-robot/).
• 🌐 Client-Side WebGPU Demos & Chrome Extension: Added 5 zero-server browser demos built on @litertjs/core (samples/web_demos/: SAM 2.1, CLIPSeg, MoGe-2 3D, PP-OCRv5, Matcha-TTS) and a Chrome Extension for on-page OCR, TTS, and 3D photo hover.
• 🧪 End-to-End Model Conversion Cookbook & Recipes: Added models/conversion.md and reproducible conversion/verification pipelines under models/ for MiniCPM5-2B, Hunyuan Hy-MT2-1.8B, Nemotron-3-Nano-4B, Bonsai Image 4B, Qwen3-TTS, SAM 3, Zipformer ASR, wav2vec2 KWS, and GLiNER2.5.
• 🏆 Real-Device Cloud Benchmarking & AI Agent Skills: Added Google Cloud Developer Device Platform (DDP) Colab notebooks, the LiteRT Performance Leaderboard, and 9 end-to-end SKILL.md playbooks (skills/) for AI coding agents.

---

🌍 The Google AI Edge & LiteRT Ecosystem

litert-samples connects the entire google-ai-edge open-source stack with the Hugging Face litert-community model ecosystem:

flowchart LR
    subgraph Source["1. Author & Convert"]
        PT["PyTorch / JAX / TF& HF Safetensors"] --> TORCH["litert-torch &ai-edge-quantizer"]
        REC["litert-samples/models/Conversion Recipes"] -.->|Scripts & Gates| TORCH
    end

    subgraph Hub["2. Model Hub (Hugging Face)"]
        TORCH -->|".tflite & .litertlm"| HF["🤗 litert-community400+ Ready-to-Run Models"]
    end

    subgraph Runtime["3. On-Device Runtimes & CLI"]
        HF --> RT["LiteRT RuntimeCompiledModel & Tensor API"]
        HF --> LM["LiteRT-LM EngineLLMs, VLMs & Tool Calling"]
        HF --> CLI["LiteRT-CLI (litert)1-Command Run & Benchmark"]
    end

    subgraph Silicon["4. Hardware Acceleration"]
        RT & LM --> HW["CPU (XNNPACK)GPU (ML Drift / OpenCL / Metal / WebGPU)NPU (Qualcomm, Google Tensor, MediaTek, Intel)"]
    end

    subgraph Apps["5. Apps, Web & Benchmarks (This Repo)"]
        HW --> SAMP["📱 samples/ • 🌐 web_demos/🏆 benchmark/ • 🤖 skills/"]
    end

Ecosystem Repositories at a Glance

---

🤗 Hugging Face litert-community Hub

Every sample application, web demo, and recipe in this repository is tightly integrated with Hugging Face litert-community — no manual conversion is required to get started, and no heavy model weights are bloated into git history.

🧭 Discover & Compare Models Instantly

🌟 Featured Model Families on litert-community

💡 Browse by Hardware & Platform Collections on Hugging Face: Android • iOS • Web LLM • Web Classical • Desktop • Google Tensor TPU • Qualcomm NPU

---

🚀 Getting Started: From Zero to Production

We organized this repository progressively so you can start at the depth that matches your goal:

1. Level 1 — Zero Install (30s): Run client-side WebGPU/WASM demos right in your browser.
2. Level 2 — 3-Minute CLI Quickstart: Download, run, and benchmark any litert-community model in 3 commands.
3. Level 3 — Run & Build Sample Apps: Build full apps for Android, iOS, Flutter, macOS, Windows, Linux, Web, and Raspberry Pi.
4. Level 4 — Convert & Quantize Models: Turn Hugging Face / PyTorch checkpoints into verified .litertlm and .tflite artifacts.
5. Level 5 — Agent Skills & Cloud Device Farm: Automate the entire lifecycle with AI coding agents and benchmark on real cloud phones.

---

🌐 Level 1 — Try in Your Browser (Zero Install)

Experience LiteRT running 100% client-side in your browser via LiteRT.js (WebGPU + multi-threaded WASM) with weights streamed directly from litert-community:

• Interactive Samples Portal (google-ai-edge.github.io/litert-samples): Unified showcase of web demos and interactive playgrounds.
• Client-Side Web Demos (samples/web_demos/) (also available on HF Spaces Demos):
  • 🖱️ SAM 2.1 Click-to-Segment (~60 ms encoder / ~4 ms per click on WebGPU) — Source
  • 🎯 CLIPSeg Open-Vocabulary Segmentation (Segment any photo by typing text prompts) — Source
  • 🧊 MoGe-2 Photo-to-3D Point Cloud (Monocular geometry rendered in interactive Three.js) — Source
  • 📄 PP-OCRv5 Document & Screenshot OCR (Text detection + recognition in browser) — Source
  • 🗣️ Matcha-TTS Neural Speech Synthesis (Flow-matching TTS + HiFi-GAN vocoder at 22 kHz) — Source
  • 🧩 LiteRT Chrome Extension: Run MoGe-2 3D hover, PP-OCRv5, and Matcha-TTS on any webpage you browse.
• C++ Tensor API Web Playground (samples/tensor_api_playground/): Interactive browser sandbox demonstrating C++ LiteRT Tensor API programs compiled to WebAssembly (Gemma 3, Image Segmentation, Mandelbrot, Game of Life).

---

⚡ Level 2 — Run & Benchmark from the Terminal (3 Minutes)

Use LiteRT-CLI (litert) or LiteRT-LM (litert-lm) to pull any model from litert-community and run or benchmark it immediately:

# 1. Install LiteRT-CLI and LiteRT-LM
pip install litert-cli-nightly

# 2. Run an LLM (.litertlm) from Hugging Face litert-community
litert download litert-community/Qwen3-0.6B
litert lm run Qwen3-0.6B.litertlm --prompt "Explain on-device AI in two sentences."

# 3. Benchmark any .tflite or .litertlm model locally or on real cloud phones (--ddp)
litert download litert-community/efficientnet_b1
litert benchmark efficientnet_b1.tflite --gpu

☁️ No phone attached? Run benchmarks on real physical Android & iOS devices in Google Cloud’s Developer Device Platform (DDP) directly from Colab — see benchmark/developer_device_platform/.

---

📱 Level 3 — Explore Sample Applications

All runnable applications live under samples/. New applications use the LiteRT CompiledModel API (for .tflite models with automatic GPU/NPU acceleration and zero-copy buffers) and LiteRT-LM (for .litertlm generative models).

🌟 Flagship Multi-Model & Cross-Platform Apps

⚡ NPU & TPU Hardware-Accelerated Samples

🎨 Vision, Speech, Audio & Text Task Samples (samples/litert/)

🖥️ Dedicated Hardware & Platform Setup Guides (samples/platforms/)

Step-by-step instructions to build, deploy, and run LiteRT & LiteRT-LM demos on specific hardware targets:

• 🤖 Android Phones (samples/platforms/android/) — GPU, CPU, and Snapdragon NPU setup.
• 🍎 iPhone / iOS (samples/platforms/ios/) — Xcode signing and Metal GPU acceleration.
• 🟢 NVIDIA Jetson Orin (samples/platforms/nvidia/) — Edge Linux GPU deployment.
• 🔌 Qualcomm / Arduino VENTUNO Q (samples/platforms/qualcomm/ventuno/) — Development board NPU/CPU guide.
• 🍓 Raspberry Pi 5 (samples/platforms/raspberry/) — Embedded Linux setup for voice & vision agents.
• 💻 Intel Core Ultra PCs, Windows (samples/platforms/intel/) — Windows build with the Intel NPU, GPU and CPU.

---

🧪 Level 4 — Convert & Quantize Models (models/)

Want to bring a new Hugging Face checkpoint to LiteRT and publish it to litert-community? Start with the Model Conversion Cookbook (models/conversion.md).

Two Conversion Lanes + C++ Tensor API

Worked Model Recipes in models/

Each directory under models/ is a complete, reproducible recipe containing export scripts, numerical parity checks against the source checkpoint, and published weights on Hugging Face:

---

🤖 Level 5 — Automate with AI Agent Skills & DDP Benchmarks

1. AI Coding Agent Skills (skills/)

Point your AI coding agent (Claude Code, Gemini CLI, Cursor, etc.) at skills/ to automate model conversion, GPU surgical patching, quantization, verification, benchmarking, and Android app generation.

• Model Lifecycle Skills (in execution order):
  1. skills/litert-conversion-workflow/ — Convert HF LLM/VLM checkpoints into verified .litertlm bundles (architecture classification, export, chat templates, quality gates).
  2. skills/gpu-clean-conversion/ — Convert PyTorch/HF models into 100% GPU-resident .tflite graphs using utilities/litert_gpu_toolkit/.
  3. skills/accuracy-safe-quantization/ — Quantize (fp16 / int8 / int4) with ai-edge-quantizer while enforcing numerical parity against float reference.
  4. skills/on-device-verification/ — Verify GPU residency, numerical accuracy, and multi-turn stability on physical devices.
  5. skills/benchmark-on-ddp/ — Benchmark models on Google Cloud Developer Device Platform (DDP) phones or macOS and update the leaderboard.
  6. skills/compiled-model-app-scaffolding/ — Scaffold a production Android app (Kotlin, Jetpack Compose, CompiledModelRunner) around a verified model.
  7. skills/litert-compiled-model-migration/ — Migrate legacy TensorFlow Lite (Interpreter) Android apps to LiteRT CompiledModel V2 with zero-copy buffers and NPU JIT.
• Android App Developer Skills (android/skills format, device-tested via skills/tests/):
  • skills/litert-runtime/ — Build an Android app running a .tflite model on CPU/GPU via CompiledModel.
  • skills/litert-lm/ — Build an Android app running a .litertlm model from litert-community with streaming multi-turn chat.

2. Real-Device Cloud Benchmarking (benchmark/)

Don’t have a fleet of test phones? Run reproducible benchmarks on real phones in Google Cloud’s Developer Device Platform (DDP) straight from your browser:

---

📂 Repository Structure

litert-samples/
├── samples/                        # 📱 Runnable sample applications across platforms
│   ├── litert/                     # Core LiteRT CompiledModel API & NPU/GPU samples (Android, iOS, Flutter, C++, Python)
│   ├── litert_model_zoo/           # 34-task Android Model Zoo app streaming on-demand from HF litert-community
│   ├── litert_lm/                  # LiteRT-LM agentic showcases (Offline Android Voice Assistant, Raspberry Pi Reachy Robot)
│   ├── web_demos/                  # Client-side WebGPU/WASM demos (@litertjs/core) + Chrome Extension
│   ├── tensor_api_playground/      # Interactive browser playground for the C++ LiteRT Tensor API
│   ├── platforms/                  # Hardware setup guides (Android, iOS, NVIDIA Jetson, Qualcomm VENTUNO Q, Raspberry Pi, Intel PCs)
│   ├── end_to_end/                 # Full conversion-to-deployment pipelines (ImageNet)
│   └── litert_interpreter/         # Legacy TensorFlow Lite Interpreter API reference samples (maintenance mode)
├── models/                         # 🧪 End-to-end model conversion recipes & verification gates
│   ├── conversion.md               # Step-by-step Model Conversion Cookbook (.litertlm & .tflite)
│   └── //           # Reproducible export, quantization & verification recipes linked to litert-community
├── skills/                         # 🤖 SKILL.md playbooks for AI coding agents (convert, quantize, verify, benchmark, scaffold)
├── benchmark/                      # 🏆 Real-device performance benchmarking suite
│   ├── leaderboard/                # Live performance leaderboard UI & measurement datasets (Android, iOS, macOS)
│   ├── developer_device_platform/  # Colab notebooks for benchmarking on Google Cloud DDP real phones
│   ├── ios/                        # Xcode / XCTest benchmark framework for local & DDP iPhones
│   └── driver/                     # Matrix runners and result collectors for building leaderboard rows
├── community/                      # 🤗 Source apps for Hugging Face litert-community Spaces
│   ├── models_page/                # Powers huggingface.co/spaces/litert-community/find-your-models
│   └── trending_page/              # Powers huggingface.co/spaces/litert-community/trending-models
└── utilities/                      # 🛠️ Shared libraries & conversion toolkits
    ├── common/                     # Shared Kotlin helpers for Android (CompiledModelRunner, camera, audio, tensor math)
    └── litert_gpu_toolkit/         # PyTorch pre-conversion surgical patches & post-conversion GPU residency checker

---

🛑 AI Coding Directives (For Developers & Coding Agents)

When authoring new application or inference code with LiteRT:

• ✅ MUST USE: The LiteRT CompiledModel API for .tflite models (Kotlin, C++, Swift, Python, JS) and LiteRT-LM for .litertlm LLM/VLM bundles.
• ❌ DO NOT USE: The legacy TensorFlow Lite Interpreter API (tflite::Interpreter, org.tensorflow.lite.Interpreter, tf.lite.Interpreter) or manual delegate wiring. Samples under samples/litert_interpreter/ are preserved only for legacy migration reference.
• 🔄 MIGRATING: Follow the LiteRT Migration Guide or run the litert-compiled-model-migration agent skill.

---

📚 Official Documentation & Resources

• LiteRT Developer Docs: ai.google.dev/edge/litert
• LiteRT-LM Developer Docs: ai.google.dev/edge/litert-lm
• Unified NPU Acceleration: ai.google.dev/edge/litert/next/npu
• LiteRT for Web (LiteRT.js): ai.google.dev/edge/litert/web
• Hugging Face Community Hub: huggingface.co/litert-community

---

🤝 Contributing & License

Contributions of new sample apps, model conversion recipes, and community models are welcome!

1. Read CONTRIBUTING.md and GOVERNANCE.md.
2. To contribute a converted .tflite or .litertlm model, follow the Model Conversion Cookbook and publish to Hugging Face litert-community.
3. Licensed under the Apache License 2.0 — see LICENSE for details.

Disclaimer: This is a sample repository maintained by Google. It is provided “as is” without warranty of any kind.