Ternary Bonsai 2 27B: A 54 GB Model in a 5.9 GB File

by Open Source Factory · September 18, 2026

On Thursday, a small Pasadena lab called PrismML put a file on Hugging Face that shouldn't really exist: a 27-billion-parameter model — vision, tool calling, 262K context — compressed into 5.9 GB. The announcement tweet did 2.8 million views overnight. The repo passed 400,000 downloads in under two days, and my entire feed is suddenly bonsai trees.

I have history with this model's family. In August I wrote about the base model, Qwen3.8-27B, and closed that post promising to revisit it "once independent numbers land and GGUF support appears." Both things just happened — sort of. GGUF support appeared, through PrismML's own llama.cpp fork (the catch is real, more below), and the numbers are in, including one the company prints in its own fine print.

So here's the honest version up front: this is the best execution of extreme low-bit compression I've seen, and the 98.2% retention claim survives a careful read — as an average. On the hardest agentic-coding benchmarks, the model keeps about three quarters of its full-precision counterpart. That is still remarkable for a file a ninth of the size. It is also not the number in the headline, and to PrismML's credit, the whitepaper says so in plain text.

So what is a ternary model, and why is this one different?

Ternary means every weight is one of exactly three values: -1, 0, or +1. Three states carry log₂3 ≈ 1.58 bits of information each, and with one shared FP16 scale per group of 128 weights, the whole language model lands at 1.76 bits per weight as shipped. The result: a model that takes about 54 GB in FP16 fits in a 5.93 GB file. For scale, the same base model needs roughly 15 GB at 4-bit and 28 GB at 8-bit.

What makes Bonsai different isn't the packing — it's the order of operations. Most "2-bit" builds (IQ2_XXS, Q2_K) are quantized after training: the model learns in full precision, then its finished weights get rounded into low-bit slots, and the damage shows up first exactly where it hurts — reasoning and long-context coding. PrismML constrains the weights to {-1, 0, +1} during training, so the model learns to live inside the constraint. Two refinements sit on top: 26.2 million parameters — under a tenth of a percent, mostly the recurrent state path of the linear-attention layers — stay in full precision, and every weight matrix is rotated through a Hadamard transform (block size 1,024) that spreads outliers around before the ternary assignment.

The difference in one picture: compress after training vs. constrain during it. The whitepaper's own summary of why conventional low-bit fails is worth framing: "the failure is easy to miss in casual testing and expensive to discover in production" (diagram: self-rendered, from PrismML's whitepaper).

The rest of the spec reads like what it is — the base model, re-plumbed. 27.36B parameters total (0.46B of them in the vision tower), Qwen3.8-27B architecture unchanged, 262K-token context, text and image in, Apache 2.0 out the door. Vision ships as a separate 0.63 GB pack you can skip if you only want text.

Item	Specification
Base model	Qwen3.8-27B (hybrid attention, architecture unchanged)
Weights	Ternary {-1, 0, +1}, FP16 group scales (g128), Hadamard-rotated basis
Effective size	1.76 bits per weight as shipped
Footprint	5.93 GB (PTQ1_0) · 7.25 GB (PQ2_0) · ~8.5 GB (Apple MLX)
Context	262K tokens
License	Apache 2.0
Runs on	PrismML's llama.cpp fork (CUDA / Metal; Vulkan, ROCm, CPU in the demo) · stock MLX on Mac

The numbers, and the asterisk

The benchmark suite covers 20 benchmarks across knowledge, math, coding, instruction following, vision and tool use, all in thinking mode. Bonsai 2 averages 83.9 against the full-precision model's 85.4 — 98.2% retention — and edges out the older Qwen3.6-27B (83.6) outright. It beats full precision on instruction following (82.66 vs 81.25), math lands within half a point, coding is level, and agentic tool calling holds at 77.57.

Where a 5.9 GB file keeps 98.2% of a 54 GB model — and where it even wins (instruction following, 102%). Vendor-reported numbers; the per-benchmark detail is in the whitepaper (chart: self-rendered from PrismML's table).

Now the asterisk. Two benchmarks that matter most to anyone building agents — Terminal-Bench 2.1 and SWE-bench Verified — tell a different story: 52.8 against 69.7, and 60.8 against 80.6. The whitepaper's own wording: "retaining roughly three quarters of the full-precision performance on both benchmarks." That is genuinely good for a file this small, and it is also a 15-to-25-point drop that the 98.2% average completely conceals. A hands-on review this week found the same thing at the user level — simple coding and vision tasks worked, while long-horizon agentic jobs (building a browser OS, a full game from scratch) sent the model into rewrite loops. The reviewer's closing take is the fair one: it is not "an Opus at home," and it doesn't need to be — "the fact that these models can actually produce lucid code, can make tool calls and function in an agentic harness over a longer context is fantastic."

The fine print, visualized. Both charts are vendor data — this is the second one you should look at (chart: self-rendered from PrismML's whitepaper).

Can you actually run it? Mostly yes — with a catch

File sizes first. The dense trit packing (PTQ1_0) is 5.93 GB. PQ2_0 is 7.25 GB — a simpler 2-bit representation that is cheaper to unpack and usually faster on big GPUs. The Apple MLX build is ~8.5 GB because MLX's container stores an extra FP16 bias per group. For context, the conventional "2-bit" IQ2_XXS build of the same base model sits around 9.4 GB at 2.8 bits per weight — more bits, bigger file.

One model, eight file sizes. The ternary packings (orange) land below even the conventional 2-bit floor — and FP16 at 53.8 GB is where this story started (chart: self-rendered; sources in the caption footer).

The catch: Bonsai 2's weights live in a rotated basis, so the runtime needs a matching Walsh-Hadamard transform — which stock llama.cpp doesn't have yet. PrismML ships its own fork, and the upstream patch is still an open pull request. The docs are refreshingly blunt about the failure modes: on a stock build the new packings are refused outright, and if you round-trip the weights into a plain Q2_0 file, "it loads silently and outputs gibberish." The MLX build runs on stock MLX, so Mac users skip the entire problem.

Assume you have the fork — the payoff is real. About 143 tokens/second on an RTX 5090, 46.8 on an M5 Max laptop, ~30 on an Nvidia L4 drawing 72 W. Energy measured at 0.714 mWh/token on an RTX 4090 — 40% better than a full-precision 8B model, which is the number that decides whether an agent can run in the background all day. In practice, a 12–16 GB single GPU or any recent Apple Silicon Mac holds the model with room for KV cache.

Decode speed across the whitepaper's hardware table, plus one community data point for scale (an M3 Max measured 19.5 tok/s). A 72-watt datacenter card doing ~30 tok/s of 27B-class output is the quiet headline here (chart: self-rendered).

And if you're wondering about older hardware: the repo's community benchmark wall has entries from the previous Bonsai generation on cards you wouldn't expect — an 11 GB GTX 1080 Ti held the whole 27B fully in VRAM at ~20 tokens/s, on a card where a conventional Q4 build of the same base model (~15.4 GB) wouldn't fit at all. Nobody has submitted a 1080 Ti run of Bonsai 2 yet, but 5.9 GB is comfortably inside an 11 GB footprint, which is a fun sentence about a 2017 GPU.

Where it lives — the GGUF repo had 400,000+ downloads and 600+ likes when I checked, under two days after upload (image: Hugging Face model card / prism-ml).

Getting it running is two commands in PrismML's demo repo: ./setup.sh fetches the right binaries and a default 7.25 GB download, then ./scripts/start_llama_server.sh gives you an OpenAI-compatible endpoint with chat, vision and tool calling on localhost:8080. The repo even ships an AGENTS.md that tells a coding agent how to set the whole thing up for you — which is either delightful or terrifying, depending on your week.

PrismML — a Caltech-spinout team in Pasadena, backed by Khosla Ventures, Cerberus and Google, with continuing support from Samsung (logo: PrismML).

The company framing for all of this is what they call "intelligence density" — how much usable capability fits per gigabyte. CEO Babak Hassibi's line is that "powerful models do not have to be confined to cloud infrastructure"; advisor Ion Stoica (UC Berkeley) adds that if the quality gap keeps closing, "it can fundamentally expand where highly capable models can be deployed." Marketing, sure — but the 95%-to-98.2% jump between two releases is the kind of number that makes a thesis credible.

So should you actually use it?

Depends on what you're doing. If you want a capable local assistant on a laptop — drafting, summarizing, document Q&A, screenshot understanding, moderate agentic loops — this is now the default answer at the 27B class, and it does it at a fraction of the energy of anything comparable. If you're serving, a single consumer GPU or a 72 W L4 at ~30 tok/s is a deployment point that didn't exist last month.

If you already run the base model at Q4 on a 24 GB card, your build is still marginally the better one — PrismML's own panel puts Bonsai 2 within 0.4 points of UD-Q4_K_XL at a third of the size. The pitch isn't "better than Q4." It's "Q4-class, on very little hardware, anywhere."

And if your workload is frontier-level agentic coding — the 15-to-25-point drop zone — the vendor's own suggestion is the sane architecture: hybrid. Run the high-frequency, privacy-sensitive steps locally, escalate the hardest few percent to a cloud model. That's also, not coincidentally, where this whole trend is headed.

My take

Two months ago, the first Bonsai 27B kept 95% of its base model. This one keeps 98.2%. Three points of retention per release is a pace the quantization field has not seen, and it's the trajectory — more than any single number — that makes the density thesis worth taking seriously. A 27B model in 5.9 GB is not a toy version of a big model. It's the big model, re-plumbed.

The fork requirement is today's tax, and I'd bet on it being temporary: the moment the kernels land upstream, the last real reason not to run this evaporates. What won't evaporate quietly: no independent full-suite rerun exists yet, and the long-horizon agentic gap is a real 20-point-ish haircut. Judge it the way the company asks — a deployment unlock, not a frontier contender — and it delivers exactly what it promises, with the fine print printed on the box.

In August I promised to come back when independent numbers and GGUF support appeared. Both showed up in their own way, so here I am — and I'm setting the next bar on the two things that are genuinely pending: the kernels landing in mainline llama.cpp, and one neutral party rerunning the suite. Expect a post when either happens.

— Figures are vendor-reported unless otherwise noted, as of September 18, 2026. Cost-adjacent and performance figures come from PrismML's announcement, whitepaper and model cards; community/independent numbers are labeled where used. Benchmark suites are the vendor's own selection; treat cross-model comparisons accordingly.

Sources:
• PrismML — Introducing Bonsai 2 27B announcement · whitepaper (PDF)
• PR Newswire — PrismML launches Bonsai 2 27B (Sep 17, Pasadena)
• Hugging Face — GGUF repo · MLX repo · model docs
• GitHub — Bonsai-demo (quickstart, community benchmarks)
• OrcaRouter — Ternary Bonsai 2 vs IQ2_XXS analysis · oMLX — M3 Max community benchmark
• YouTube — "Bonsai 2 27B First Test" hands-on (Sep 18) · the July Bonsai 27B release for comparison
• Charts: self-rendered from PrismML whitepaper tables · images: PrismML (logo), Hugging Face (model card)