AirLLM uses “Layer-wise Inference” to run large language models on consumer GPUs. Instead of loading the entire model into VRAM, it loads, computes, and flushes one layer at a time.

Key features:

• Run 70B models on a single 4GB GPU
• Scales to Llama 3.1 405B on just 8GB VRAM
• No quantization needed by default
• Supports Llama, Qwen, and Mistral
• Works on Linux, Windows, and macOS
• 100% Open Source

Caveat from users: speed is limited by disk I/O — expect around 1-2 tokens/second since layers are loaded from disk rather than VRAM.

Source: https://www.instagram.com/p/DV-7c-vj70_/