I replaced MiniMax H3's 32B text encoder with a 4B one — on an 11GB card
I replaced MiniMax H3's 32B text encoder with a 4B one — on an 11GB card
Every time I run MiniMax H3 on my 1080Ti, I watch the same absurdity: a 20GB video model being conditioned by a 15.7GB text encoder that exists only to turn a prompt into a tensor. Then someone on X posted a project called ClipProj that claims to replace that 32B encoder with a 4B one using — I had to read this twice — a single learned linear matrix. So I installed it, verified it, and it actually works.
What's actually being replaced
MiniMax H3 doesn't use a small CLIP-style encoder like older video models. It conditions on Qwen3-VL-32B truncated to 50 layers, in NVFP4 — 15.7GB of weights whose only job is converting your prompt into a [seq, 5120] conditioning tensor. On an 11GB card that means constant offloading. The encoder is the bottleneck of the whole pipeline, not the video model itself.
ClipProj (MIT-licensed, by NicoLab28) replaces it with Qwen3-VL-4B — 4.9GB — plus one learned 2560→5120 linear map. It works because both models share the same tokenizer: the same prompt produces the same tokens at the same positions in both models, so a position-by-position mapping between their hidden states is learnable. No alignment problem, no training run — the calibration is plain ridge regression. Encode N prompts with both models, accumulate the statistics, solve. That's it.
The authors were refreshingly honest about the numbers: test cosine 0.71, R² 0.51, and an explicit "this is a proof of concept." A cosine of 0.71 sounds bad until you learn the DiT tolerates far more error than the metric suggests. I wasn't going to trust any of it — I have a GPU and a test prompt.
The verification: "a red ball on a wood table"
The project ships two control matrices alongside the real one, specifically so you can prove the learned matrix is doing the work and not the diffusion model hallucinating compliance. Same prompt, same seed, only the matrix changes:
| Matrix | What "a red ball on a wood table" produced |
|---|---|
| Zero control (W = 0, no prompt info) | A countryside landscape — prompt completely ignored |
| Identity control (raw 4B dims, no learning) | A golden object in flames — unusable |
| Learned matrix | A red ball on a wood table |
The loader log printed cos_test 0.8450 on my copy of the matrix — the projection math is real, not a stub. Controls fail as designed, the learned matrix succeeds. That's the whole experiment in one table.
The speed rabbit hole (this is where it got interesting)
My first headless run was a disaster: 18 minutes per sampling step at a modest resolution. That's ~10x slower than my normal H3 runs, and I spent hours chasing it — GPU clocks fine, no throttling, same install, same workflow shape. The culprit turned out to be which VRAM mode ComfyUI picked at model-load time, and it split into two separate discoveries:
- Normal mode (async weight offloading): fast — 43s per step at 0.1MP. But the turbo LoRA fails to patch: ERROR lora ... Allocation on device on a dozen blocks every step. The video would have been generated WITHOUT the LoRA — invalid turbo output.
- Lowvram mode: the mode my Desktop app has been using all along. LoRA applies cleanly, output is valid — at ~151s per step, which matches my usual pace.
The irony: the 4B encoder being CPU-resident gives the DiT more VRAM, which flips ComfyUI into the fast mode — which then breaks LoRA patching. So the fix for speed is the thing that breaks correctness, at least until someone fixes the allocation path. I left the server in lowvram mode and moved on. (If the LoRA issue gets fixed, the 4B swap becomes a genuine ~4x speedup on this card — that's the open thread.)
Where you lose things (they told you, I confirmed)
1. Named references vanish. The 4B stores far fewer facts than the 32B. In the author's tests one famous actor rendered correctly while another became a generic figure. No projection can restore knowledge that was never encoded. Fix: describe instead of name — "the actress X as [role], blonde, blue eyes" survives; the bare name often doesn't.
2. Non-English speech degrades. The audio branch of the DiT is far more demanding than the video branch. English lines reconstruct fine; French (their test) came out phonetically broken. Korean voice work would be the first thing I'd A/B before trusting this for real projects.
3. It's a proof of concept. Tested by the author on one Windows machine, now on mine. Breaking changes, rough edges, and the projection is at its ceiling — the author measured that 8x more calibration data bought only 1.8% more cosine. The next step is an MLP, not more prompts.
Should you care?
If you run H3 on a card where the encoder offloading hurts — 11GB here, but also 12GB/16GB cards where the 32B just barely doesn't fit — yes. The 4.9GB encoder + projection keeps your VRAM for the actual video model, and the encode step itself gets dramatically faster. If you're on a 24GB+ card, this solves a problem you don't have; the named-reference and non-English losses aren't worth it.
The part I respect most is the control matrices. A random GitHub project that ships a way to falsify its own claim, runs the identity control, and tells you what "unusable" looks like — that's how you earn trust in this corner of the internet. The red ball on my wood table came out glossy. I'm going to keep this installed and run it against my real Korean-voice prompts before I trust it for production work.
If you try it: put the .safetensors matrices in ComfyUI/models/clip_projections/, load the 4B encoder on CPU with dynamic mode, and run the zero/identity controls first — the README walks through it. Project: github.com/nicolab28/ComfyUI-ClipProj.
All numbers measured on my machine (1080Ti 11GB, ComfyUI 0.30.2, turbo LoRA 10 steps), Aug 10 2026. The MiniMax H3 setup post is on this blog if you want the baseline.
Related on this blog: MiniMax Music 3: Five-Minute Songs, Open Weights, One $20M Catch · MiniMax H3 went open-weight, so I did the math on whether my PC can run it
Comments
Post a Comment