MODELS & COSTS

Bonsai 2 27B: A Smaller File With Task-Level Trade-offs

Open Source Factory · Sep 18, 2026 · 8 min read

A roughly 5.9GB model file is easier to place than 54GB of raw half-precision weights. The saving does not preserve every task equally. Bonsai 2's weaker coding rows and its required runtime fork are worth checking before turning the compact download into a local deployment plan.

In this article

1. Ternary weights and storage overhead
2. Average retention and the weaker tasks
3. The supported runtime fork
4. Choosing for a particular local task

Maintenance, October 17, 2026: a broken reference link has been replaced. Pricing and availability in this article remain the dated snapshot stated in the text; a current documentation link does not turn historical figures into a current offer.

I have history with this model's family. In August I wrote about the base model, Qwen3.8-27B, and closed that post promising to revisit it "once independent numbers land and GGUF support appears." Both things just happened — sort of. GGUF support appeared, through PrismML's own llama.cpp fork (the catch is real, more below), and the numbers are in, including one the company prints in its own fine print.

PrismML’s 98.2% retention figure is an average over its evaluation set. Terminal-Bench 2.1 and SWE-bench Verified retain closer to three quarters of the full-precision scores. A file roughly one-ninth the size is a storage result; whether that trade is useful depends on which capabilities the workload needs.

Ternary weights and storage overhead

Ternary means every weight is one of exactly three values: -1, 0, or +1. Three states carry log₂3 ≈ 1.58 bits of information each, and with one shared FP16 scale per group of 128 weights, the whole language model lands at 1.76 bits per weight as shipped. The result: a model that takes about 54 GB in FP16 fits in a 5.93 GB file. For scale, the same base model needs roughly 15 GB at 4-bit and 28 GB at 8-bit.

What makes Bonsai different isn't the packing — it's the order of operations. Most "2-bit" builds (IQ2_XXS, Q2_K) are quantized after training: the model learns in full precision, then its finished weights get rounded into low-bit slots, and the damage shows up first exactly where it hurts — reasoning and long-context coding. PrismML constrains the weights to {-1, 0, +1} during training, so the model learns to live inside the constraint. Two refinements sit on top: 26.2 million parameters — under a tenth of a percent, mostly the recurrent state path of the linear-attention layers — stay in full precision, and every weight matrix is rotated through a Hadamard transform (block size 1,024) that spreads outliers around before the ternary assignment.

The difference in one picture: compress after training vs. constrain during it. The whitepaper's own summary of why conventional low-bit fails is worth framing: "the failure is easy to miss in casual testing and expensive to discover in production" (diagram: self-rendered, from PrismML's whitepaper).

The rest of the spec reads like what it is — the base model, re-plumbed. 27.36B parameters total (0.46B of them in the vision tower), Qwen3.8-27B architecture unchanged, 262K-token context, text and image in, Apache 2.0 out the door. Vision ships as a separate 0.63 GB pack you can skip if you only want text.

Item	Specification
Base model	Qwen3.8-27B (hybrid attention, architecture unchanged)
Weights	Ternary {-1, 0, +1}, FP16 group scales (g128), Hadamard-rotated basis
Effective size	1.76 bits per weight as shipped
Footprint	5.93 GB (PTQ1_0) · 7.25 GB (PQ2_0) · ~8.5 GB (Apple MLX)
Context	262K tokens
License	Apache 2.0
Runs on	PrismML's llama.cpp fork (CUDA / Metal; Vulkan, ROCm, CPU in the demo) · stock MLX on Mac

Average retention and the weaker tasks

The benchmark suite covers 20 benchmarks across knowledge, math, coding, instruction following, vision and tool use, all in thinking mode. Bonsai 2 averages 83.9 against the full-precision model's 85.4 — 98.2% retention — and edges out the older Qwen3.6-27B (83.6) outright. It beats full precision on instruction following (82.66 vs 81.25), math lands within half a point, coding is level, and agentic tool calling holds at 77.57.

Where a 5.9 GB file keeps 98.2% of a 54 GB model — and where it even wins (instruction following, 102%). Vendor-reported numbers; the per-benchmark detail is in the whitepaper (chart: self-rendered from PrismML's table).

Now the asterisk. Two benchmarks that matter most to anyone building agents — Terminal-Bench 2.1 and SWE-bench Verified — tell a different story: 52.8 against 69.7, and 60.8 against 80.6. The whitepaper's own wording: "retaining roughly three quarters of the full-precision performance on both benchmarks." That is good for a file this small, and it is also a 15-to-25-point drop that the 98.2% average completely conceals. A hands-on review this week found the same thing at the user level — simple coding and vision tasks worked, while long-horizon agentic jobs (building a browser OS, a full game from scratch) sent the model into rewrite loops. The reviewer's closing take is the fair one: it is not "an Opus at home," and it doesn't need to be — "the fact that these models can actually produce lucid code, can make tool calls and function in an agentic harness over a longer context is fantastic."

The fine print, visualized. Both charts are vendor data — this is the second one you should look at (chart: self-rendered from PrismML's whitepaper).

The supported runtime fork

File sizes first. The dense trit packing (PTQ1_0) is 5.93 GB. PQ2_0 is 7.25 GB — a simpler 2-bit representation that is cheaper to unpack and usually faster on big GPUs. The Apple MLX build is ~8.5 GB because MLX's container stores an extra FP16 bias per group. For context, the conventional "2-bit" IQ2_XXS build of the same base model sits around 9.4 GB at 2.8 bits per weight — more bits, bigger file.

One model, eight file sizes. The ternary packings (orange) land below even the conventional 2-bit floor — and FP16 at 53.8 GB is where this story started (chart: self-rendered; sources in the caption footer).

The catch: Bonsai 2's weights live in a rotated basis, so the runtime needs a matching Walsh-Hadamard transform — which stock llama.cpp doesn't have yet. PrismML ships its own fork, and the upstream patch is still an open pull request. The docs are refreshingly blunt about the failure modes: on a stock build the new packings are refused outright, and if you round-trip the weights into a plain Q2_0 file, "it loads silently and outputs gibberish." The MLX build runs on stock MLX, so Mac users skip the entire problem.

Assume you have the fork — the payoff is real. About 143 tokens/second on an RTX 5090, 46.8 on an M5 Max laptop, ~30 on an Nvidia L4 drawing 72 W. Energy measured at 0.714 mWh/token on an RTX 4090 — 40% better than a full-precision 8B model, which is the number that decides whether an agent can run in the background all day. In practice, a 12–16 GB single GPU or any recent Apple Silicon Mac holds the model with room for KV cache.

Decode speed across the whitepaper's hardware table, plus one community data point for scale (an M3 Max measured 19.5 tok/s). A 72-watt datacenter card doing ~30 tok/s of 27B-class output is the quiet headline here (chart: self-rendered).

And if you're wondering about older hardware: the repo's community benchmark wall has entries from the previous Bonsai generation on cards you wouldn't expect — an 11 GB GTX 1080 Ti held the whole 27B fully in VRAM at ~20 tokens/s, on a card where a conventional Q4 build of the same base model (~15.4 GB) wouldn't fit at all. Nobody has submitted a 1080 Ti run of Bonsai 2 yet, but 5.9 GB is comfortably inside an 11 GB footprint, which is a fun sentence about a 2017 GPU.

Where it lives — the GGUF repo had 400,000+ downloads and 600+ likes when I checked, under two days after upload (image: Hugging Face model card / prism-ml).

Getting it running is two commands in PrismML's demo repo: ./setup.sh fetches the right binaries and a default 7.25 GB download, then ./scripts/start_llama_server.sh gives you an OpenAI-compatible endpoint with chat, vision and tool calling on localhost:8080. The repo even ships an AGENTS.md that tells a coding agent how to set the whole thing up for you — which is either delightful or terrifying, depending on your week.

PrismML — a Caltech-spinout team in Pasadena, backed by Khosla Ventures, Cerberus and Google, with continuing support from Samsung (logo: PrismML).

The company framing for all of this is what they call "intelligence density" — how much usable capability fits per gigabyte. CEO Babak Hassibi's line is that "powerful models do not have to be confined to cloud infrastructure"; advisor Ion Stoica (UC Berkeley) adds that if the quality gap keeps closing, "it can fundamentally expand where highly capable models can be deployed." Marketing, sure — but the 95%-to-98.2% jump between two releases is the kind of number that makes a thesis credible.

Choosing for a particular local task

Depends on what you're doing. If you want a capable local assistant on a laptop — drafting, summarizing, document Q&A, screenshot understanding, moderate agentic loops — this is now the default answer at the 27B class, and it does it at a fraction of the energy of anything comparable. If you're serving, a single consumer GPU or a 72 W L4 at ~30 tok/s is a deployment point that didn't exist last month.

If you already run the base model at Q4 on a 24 GB card, your build is still marginally the better one — PrismML's own panel puts Bonsai 2 within 0.4 points of UD-Q4_K_XL at a third of the size. The pitch isn't "better than Q4." It's "Q4-class, on very little hardware, anywhere."

And if your workload is frontier-level agentic coding — the 15-to-25-point drop zone — the vendor's own suggestion is the sane architecture: hybrid. Run the high-frequency, privacy-sensitive steps locally, escalate the hardest few percent to a cloud model. That's also, not coincidentally, where this whole trend is headed.

The cited retention changes from 95% for the earlier Bonsai 27B to 98.2% here. That is a reported average across selected evaluations, not proof that every capability is preserved. A smaller file is valuable when the quality on the intended task remains adequate.

The runtime fork is a current deployment requirement, and an upstream merge should not be assumed. Independent full-suite testing was also absent at the article’s date. The long-horizon coding loss means this should be assessed as a capacity/quality trade-off for a specific task.

Mainline runtime support and a neutral rerun would make the choice easier to judge. Until then, check the supported fork and the weakest relevant benchmark row before treating the file-size saving as a deployment recommendation.

— Figures are vendor-reported unless otherwise noted, as of September 18, 2026. Cost-adjacent and performance figures come from PrismML's announcement, whitepaper and model cards; community/independent numbers are labeled where used. Benchmark suites are the vendor's own selection; treat cross-model comparisons accordingly.

Sources:
• PrismML — Introducing Bonsai 2 27B announcement · whitepaper (PDF)
• PR Newswire — PrismML launches Bonsai 2 27B (Sep 17, Pasadena)
• Hugging Face — GGUF repo · MLX repo · Bonsai documentation introduction
• GitHub — Bonsai-demo (quickstart, community benchmarks)
• OrcaRouter — Ternary Bonsai 2 vs IQ2_XXS analysis · oMLX — M3 Max community benchmark
• YouTube — "Bonsai 2 27B First Test" hands-on (Sep 18) · the July Bonsai 27B release for comparison
• Charts: self-rendered from PrismML whitepaper tables · images: PrismML (logo), Hugging Face (model card)

Edited October 26, 2026. Original publication and evidence dates are retained; this edit is not a new product test.

Reading guideAbout & methodsCorrectionsPrivacy