4GB VRAM Runs 70B, 12GB Runs 671B: Dissecting AirLLM's "Layer-wise Streaming" Magic

Created2026-09-29|Updated2026-09-29|Tech Sharing

|Post Views:

Author: 棋飞

Link: https://qifeidoc.com/en/posts/airllm-layer-streaming/

Copyright Notice: All articles on this blog are licensed under CC BY-NC-SA 4.0 unless otherwise stated.

AI LLM AirLLM VRAM Optimization Inference LoRA Tutorial

Related Articles

4GB 显存跑 70B，12GB 跑 671B：拆解 AirLLM 的"分层流式"魔法

大模型圈有一类话题永远有热度：”我家显卡只有 X GB，能跑多大模型？” 以前这个问题的答案很残酷——8B 要 16GB，70B 要 80GB，405B 得上 4 卡 H100，Kimi K3 这种 2.8T 的怪物根本别想本地玩。但 GitHub 上有个叫 AirLLM 的项目，把这套心智模型掀了个底朝天：模型参数量 AirLLM 实测 VRAM Qwen3-8B / Mistral-7B 8B ~1–2 GB Qwen3-30B / Mixtral 30–47B ~1–3 GB Qwen3.8-27B (含视觉) 27B 3.33 GB Qwen3.8-Flash-Next ~180B (MoE + PLE) 5.95 GB Qwen3-235B 235B ~3 GB Llama 3.x 70B 70B ~4 GB Llama 3.1 405B 405B ~8 GB DeepSeek-V3 671B ~12 GB Kimi K3 2.8T 3.72 GB 你没看错——671B 全量化训练挂一层在 12GB...

AI 在"思考"的时候到底在做什么：一份大模型底层逻辑的拆解笔记

最近 AI 圈有一个奇怪的现象:模型越来越像”会思考的人”,但它们的底层,其实从来没变过。这一篇我想把”大模型到底是什么”这件事讲清楚。不是讲 API、不是讲 Prompt 技巧,是从最底层——它到底是怎么”吐出下一个字”的——拆给你看。看完你会发现,我们平时说的”AI 思考”、”AI 推理”,其实都不是它真的在思考。一、先把一件事说死:AI 不认识字很多人对 AI 的第一误解,就是把 ChatGPT、Claude、文心一言想象成一个”读过全人类书籍的超级大脑”。这个画面很酷,但从原理上就不对。大模型的输入和输出,从头到尾只有一种东西:数字序列,在专业术语里叫 Token(词元)。所以当你问 AI “你今天吃饭了吗”这句话,它看到的不是中文汉字,而是一串数字。比如它眼里这句话可能是: 12345你 → 123今天 → 456吃饭 → 789了 → 012吗 → 345 这个”汉字 ↔ 数字”的翻译过程,叫做分词(Tokenization),由一个叫”分词器”的小组件完成。这里有一个细节容易踩坑:同一个汉字,在不同的模型里对应的数字可能完全不同。同一句话,丢给 GPT...

Codex (ChatGPT) Six-Month Update Roundup: 4 Hard Tips to Save Tokens + 8 New Tricks

It’s been six months since our last beginner’s guide to Codex. In those six months, Codex has evolved from a standalone desktop app into a feature integrated inside the ChatGPT client, and filled out a whole set of new capabilities: Sites, the office productivity suite, App Shots, Hooks, and more. This post breaks down every key update and focuses on the most practical “token-saving toolkit” for anyone currently using — or about to use — Codex. Bottom line up front: if you only use Chat mode ...

AI Daily · 2026-10-10

昨天 Google 把通用 Gemini 工作 Agent 摆上桌面，今天 AI Daily 的关键词从「模型本身」扩散到「模型之外的全栈」：OpenAI 用 Operator 企业版正面迎战、Anthropic 用 Claude 5 Sonnet 加固 reasoning、欧盟用 AI Act 第一阶段细则开始划线，叠加 Meta、Apple、Mistral 在端侧与开源评测上的同步推进，整个行业进入「产品 + 政策 + 基础设施」三轮齐转的节奏。昨天的全景可参考 10-09 的 AI Daily。 OpenAI Operator 进入企业市场 OpenAI launches Operator for Enterprise with full API access · TechCrunch · 2026-10-09 事实：OpenAI 正式把 Operator 推向企业客户，开放完整 API 与 SSO/SCIM 集成，新增「任务审计日志」「权限分级」和「私有模型路由」三个企业级模块。背景：Operator 自去年发布以来一直停留在 ChatGPT Pro 内的 ...

AI Daily · 2026-10-09

Today’s AI circles have been almost entirely dominated by two words: “Agent” and “Google.” Gemini’s supercharged work Agent has launched, joined by product moves from Nous Research and AgentGarten at the enterprise and training-ground tiers. Meanwhile, the open-source world keeps pushing: Visa is releasing its AI defense systems, and wearable hardware is putting the Agent right at your fingertip. Looking ahead, the keyword for the second half of 2026 has clearly shifted from “the model itself...

AI Daily · 2026-10-10

Yesterday Google put a general-purpose Gemini work Agent on the desktop; today AI Daily’s keywords spread from “the model itself” to “the full stack around the model”: OpenAI meets the moment head-on with Operator for Enterprise, Anthropic hardens reasoning with Claude 5 Sonnet, the EU starts drawing lines with Phase 1 rules of the AI Act. Layered on top, Meta, Apple, and Mistral are advancing in lockstep on the on-device and open-source benchmarking fronts. The entire industry has entered a ...

数据加载中