Run 70B+ LLMs on consumer GPUs using layer streaming
概览
Run 70B+ parameter language models on consumer GPUs with 4-8GB VRAM. AirLLM uses to run massive models on hardware that shouldn't be able to run them: - Load one transformer layer at a time - Compute, discard, repeat - SSD becomes the bottleneck, not GPU memory - Batch processing overnight - One-shot generation - Proof-of-concept on consumer hardware - Swarm deployments - SKILL.md — Full documentation - example.py — Working example script - requirements.txt — Compatible dependencies If 70B models run on consumer GPUs, AI doesn't have to live in corporate data centers. Distributed intelligence without gatekeepers. - AirLLM by Gavin Li - Skill docs by @VelumKai
README
AirLLM Skill
Run 70B+ parameter language models on consumer GPUs with 4-8GB VRAM.
What is this?
AirLLM uses layer-wise streaming to run massive models on hardware that shouldn’t be able to run them:
- Load one transformer layer at a time
- Compute, discard, repeat
- SSD becomes the bottleneck, not GPU memory
Quick Start
# Install dependencies (order matters!)
pip install torch --index-url https://download.pytorch.org/whl/cu121
pip install "optimum<1.17" "transformers==4.44.0" airllm
# Run example
python example.py
Performance
| Model | GPU | Speed |
|---|---|---|
| 7B | RTX 3070 8GB | ~14s/token |
| 70B | RTX 3070 8GB | ~2min/token |
Slow, but it works.
Use Cases
- Batch processing overnight
- One-shot generation
- Proof-of-concept on consumer hardware
- Swarm deployments
Files
SKILL.md— Full documentationexample.py— Working example scriptrequirements.txt— Compatible dependencies
Why This Matters
If 70B models run on consumer GPUs, AI doesn’t have to live in corporate data centers. Distributed intelligence without gatekeepers.
Credits
License
MIT
推荐工具
换一个关键词，或者移除筛选条件。
安装
npx skillfish add velumkai/airllm-skill