Prism Ternary Bonsai 27B: Full Reasoning Model in 5.9 GB

Prism Ternary Bonsai 27B is a fully capable 27-billion-parameter reasoning model whose ternary weights occupy roughly 5.9 GB and which runs at about 26 tokens per second on a consumer laptop. Released by AI research outfit Prism ML, the prism ternary bonsai 27b gguf retains 94.6% of its FP16 baseline's benchmark quality at just 1.71 bits per weight. It challenges a foundational assumption in local AI deployment: that serious reasoning quality requires either painful quantization trade-offs or expensive hardware. This article covers the architecture, benchmark results, hardware requirements, file structure, and developer setup in full.

Fact	Value
Full name	prism-ml/Ternary-Bonsai-27B-gguf
Parameter count	~27.3 B (24.8 B backbone + 2.5 B embed/LM head)
Weight format	Natively trained ternary (−1, 0, +1) at 1.71 bpw
Base architecture	Qwen3.6-27B (64 transformer blocks, hybrid attention)
Context window	262,144 tokens
Main GGUF file size	7.17 GB (weights); ~5.9 GB weight footprint, ~7.2 GB resident at inference
Benchmark average (15 tasks)	80.49 vs. 85.07 FP16 baseline (94.6% retention)
Laptop speed (M5 Pro)	26.2 tok/s (TG128)
License	Apache 2.0
Hugging Face likes	652

What Is the Prism Ternary Bonsai Model and Where Does It Come From?

The Prism Ternary Bonsai 27B GGUF on Hugging Face is a full 27-billion-parameter language model whose weights were natively trained in ternary format — meaning every weight in the backbone is one of exactly three values: −1, 0, or +1. This is not post-training quantization (PTQ), where a full-precision model gets rounded down after training. Prism ML treats the ternary representation as a first-class training objective: the model was built to operate at 1.71 bits per weight from the outset.

The name "Bonsai" is apt. Like the horticultural practice of cultivating a full-scale tree in a fraction of its natural footprint, the Bonsai series achieves full-scale reasoning in a dramatically compressed form — not by pruning capability after the fact, but by growing it that way from the start.

The underlying architecture derives from Qwen3.6-27B, Alibaba's hybrid-attention causal language model. The backbone spans 64 transformer blocks with approximately 24.8 billion parameters; an additional 2.5 billion parameters live in the embedding and language model head layers, giving a total of roughly 27.3 billion ternary language weights. An optional 460-million-parameter (27-block) vision tower — quantized separately at HQQ 4-bit — is bundled for multimodal (vision + text) use cases. The architecture keeps Qwen3.6's core design choices intact: SwiGLU MLP activations, RoPE positional encoding, RMSNorm, and a hybrid attention scheme where only about 16 of the 64 layers use full quadratic attention while the remaining ~48 use linear attention (roughly a 75% linear / 25% full split). That hybrid design dramatically reduces KV-cache growth — a critical advantage for long-context inference.

The result is a model that supports a 262,144-token context window. Because the full-attention KV cache grows on only 16 of 64 layers, that cache stays around 4.3 GB even at the full 262K window, and with 4-bit KV quantization enabled the peak memory requirement for the full context stays well within reach of a 16 GB consumer GPU — a figure that would be impossible for a standard 27B FP16 model on consumer hardware.

How Ternary Weights Work — and Why It Matters for the Prism Ternary Model

Standard neural network weights are stored as 16-bit or 32-bit floating-point numbers. PTQ tools like llama.cpp's Q4_K or Q8_0 formats approximate those floats with lower-precision integers after training finishes — but the model was never trained to use those integers, so accuracy degrades, increasingly so as bit-width falls below 4 bits.

Ternary training takes a different path. Each weight holds a value from {−1, 0, +1}. The information-theoretic minimum for a ternary value is log₂(3) ≈ 1.585 bits per weight, but the deployed format — Q2_0_g128 — achieves 1.71 effective bits per weight by packing weights into 2-bit slots (three of the four possible 2-bit states are used, giving a 2.125-bit deployed slot cost) and storing one shared FP16 scale factor per group of 128 weights.

Relative to binary (1-bit) schemes, the extra zero state provides a more expressive weight alphabet that recovers substantially more of the full-precision model's learned behavior — which is why Prism ML positions ternary as the quality-oriented operating point of the Bonsai 27B family.

The zero state is what separates ternary from 1-bit (binary) schemes. In 1-bit models, every weight is either active (+1) or inverted (−1) — there is no silence. Ternary weights can contribute nothing, letting the network learn selective, sparse activation patterns more faithfully. Prism ML publishes both variants: the ternary model is the quality-oriented choice, while the 1-bit companion (Bonsai-27B, ~3.9 GB, 1.125 bpw) targets the "phone-class operating point." Per the model card, that ~3.9 GB 1-bit build is the one small enough to fit an iPhone 17 Pro Max via MLX Swift, whereas the ternary build's ~7.2 GB resident footprint exceeds iOS's ~6 GB per-app memory budget and is therefore laptop/GPU-only.

Crucially, the packed ternary weights are consumed directly by custom kernels and never expanded back to FP16 at runtime. The smaller memory footprint therefore translates directly into faster memory-bandwidth-limited inference — not just smaller downloads.

Benchmark Performance: How Much Quality Do You Actually Lose?

Prism ML evaluated the prism ternary bonsai using EvalScope + vLLM on an NVIDIA H100, running 15 benchmarks across six skill categories in thinking mode (temperature 0.7, top-p 0.95, top-k 20). The headline number: Ternary Bonsai 27B scores 80.49 average across all 15 benchmarks, versus 85.07 for the FP16 baseline — a 94.6% retention rate at 1.71 bits per weight.

Variant	True bpw	Footprint	Thinking Avg (15 benchmarks)	vs FP16
Qwen3.6-27B FP16	16.0	54 GB	85.07	100%
Qwen3.6-27B Q4_K_XL ("4-bit" PTQ)	5.2	17.6 GB	84.99	99.9%
Qwen3.6-27B IQ2_XXS ("2-bit" PTQ)	2.8	9.4 GB	72.73	85.5%
Gemma-4-31B FP16	16.0	61.5 GB	84.58	99.4%
Gemma-4-31B QAT ("4-bit")	6.0	23.3 GB	83.41	98.0%
Gemma-4-31B Q2_K_XL ("2-bit" PTQ)	3.0	11.8 GB	73.31	86.2%
Ternary Bonsai 27B (this model)	1.71	5.9 GB	80.49	94.6%
1-bit Bonsai 27B	1.125	3.9 GB	76.11	89.5%

The most striking comparison is against standard post-training 2-bit quantization of the same base model. The PTQ IQ2_XXS build of Qwen3.6-27B scores only 72.73 — nearly 8 points lower than the ternary model — while still needing 9.4 GB, or roughly 60% more memory. Ternary training recovers roughly two-thirds of the accuracy loss that aggressive PTQ introduces, at lower memory cost. The prism ternary model is simultaneously better and smaller than the PTQ 2-bit alternative derived from its own base model.

Category-Level Performance Breakdown

The gap between the ternary model and the FP16 baseline is not uniform across task types. Math is where the model holds up best:

Category	Benchmarks	FP16 Score	Ternary 27B Score	Gap
Math	GSM8K, MATH-500, AIME25, AIME26	95.33	93.40	−1.93
Coding	HumanEval+, MBPP+, LiveCodeBench	88.74	85.96	−2.78
Instruction Following	IFEval, IFBench	78.47	71.77	−6.70
Agentic / Tool Calling	BFCL v3, τ²-Bench	80.00	74.01	−5.99
Knowledge & Reasoning	MMLU-Redux, MuSR	83.15	76.96	−6.19
Vision	MMMU-Pro, OCR Bench v2	72.61	65.19	−7.42

Developers building math-heavy or code-generation applications will find the ternary model nearly indistinguishable from the full-precision baseline. The largest degradation shows up in vision tasks and instruction following — areas historically more sensitive to fine-grained weight precision. Within the math category, individual benchmarks such as GSM8K (arithmetic word problems) show very small gaps between the ternary and FP16 models — in some cases within the range of normal evaluation variance. That confirms the quality floor here is solid rather than fragile, even if you should treat any single per-benchmark reversal as noise rather than genuine superiority.

Hardware Requirements, VRAM, and Real-World Inference Speed

Prism ML has documented hardware requirements for the prism ternary bonsai 27b gguf concretely across three major deployment environments: Apple Silicon via Metal, NVIDIA GPUs via CUDA, and CPU-only via standard llama.cpp. The published benchmark table reports a resident memory footprint of about 7.2 GB during inference (the ~5.9 GB weight footprint plus KV cache, activations, and runtime overhead):

Platform	Memory Footprint	Token Generation (TG128)	Prompt Processing (PP512)
Apple M4 Pro (Metal)	7.2 GB	18.0 tok/s	125 tok/s
Apple M5 Pro (Metal)	7.2 GB	26.2 tok/s	393 tok/s
Apple M5 Max (Metal)	7.2 GB	44.0 tok/s	830 tok/s
NVIDIA H100 (CUDA)	7.2 GB	98.0 tok/s	2,596 tok/s
NVIDIA H100 + DSpark	~7.7 GB	131.8 tok/s	—

The 26.2 tok/s figure on an M5 Pro is the headline for laptop users — fast enough for real-time conversation, given that comfortable human reading speed sits around 4–5 words (5–7 tokens) per second. At 44 tok/s on M5 Max, the model works well for interactive coding assistants and agent loops where latency directly affects developer experience. Adding the DSpark speculative-decoding drafter on an H100 pushes throughput from 98.0 to 131.8 tok/s — a 1.34× speedup (accepted draft length τ ≈ 3.7 at draft depth k = 4) at no accuracy cost, since speculative decoding is lossless by design: accepted tokens are statistically indistinguishable from those the target model would have generated on its own.

VRAM at Extended Context Lengths

The hybrid attention design — roughly 16 full-attention layers plus ~48 linear-attention layers — pays real dividends as context length grows, because linear-attention layers add no KV-cache entries:

Build	Weights	4K ctx	10K ctx	100K ctx
Ternary Bonsai (Q2_0)	7.15 GB	8.4 GB	8.7 GB	14.7 GB
Qwen3.6-27B Q4_K_XL	17.6 GB	19.2 GB	19.6 GB	25.6 GB
Qwen3.6-27B FP16	51.25 GB	52.6 GB	53.3 GB	59.3 GB

Enabling near-lossless 4-bit KV quantization drops the 100K-context peak from 14.7 GB to roughly 10.1 GB, and — with the full-attention cache capped near 4.3 GB because it grows on only 16 of 64 layers — the full 262K-token window fits within the memory of a single 16 GB consumer GPU or a maxed-out MacBook Pro. That makes the prism ternary model a practical choice for long-document pipelines, retrieval-augmented generation (RAG) systems, and agentic workloads requiring deep context — previously the exclusive domain of multi-GPU clusters running full-precision models.

The Files: What You Actually Download

The repository ships several distinct GGUF files, composable to match your use case:

• Ternary-Bonsai-27B-Q2_0.gguf (7.17 GB) — the primary language model in Q2_0_g128 ternary format; the file most users will want first.
• DSpark drafter (Q4_1) (1.95 GB) — the default speculative-decoding drafter; contributes only ~0.5 GB of unique weights at serving precision; boosts throughput by 1.34× on H100.
• DSpark drafter (BF16) (7.29 GB) — reference-precision drafter for research or fine-tuning pipelines.
• Vision tower mmproj (HQQ 4-bit, Q8_0 container) (0.63 GB) — enables multimodal (image + text) inference via the 460M-parameter, 27-block vision tower.
• Vision tower mmproj (BF16) (0.93 GB) — reference-precision vision tower.
• Group-64 ternary pack (Q2_0_g64) (7.59 GB) — described on the card as an "also published" alternative with finer group granularity (64 weights per scale factor instead of 128), potentially recovering a small additional quality margin at a modest size cost.

The repository requires Prism ML's own llama.cpp fork, which ships the custom Q2_0_g128 hybrid-attention ternary kernels for CUDA and Metal. Running on CUDA:

git clone https://github.com/PrismML-Eng/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON && cmake --build build -j
huggingface-cli download prism-ml/Ternary-Bonsai-27B-gguf \
  Ternary-Bonsai-27B-Q2_0.gguf --local-dir .
./build/bin/llama-cli \
  -m Ternary-Bonsai-27B-Q2_0.gguf \
  -p "Explain quantum computing in simple terms." \
  -n 256 \
  --temperature 0.7 --top-p 0.95 --top-k 20 \
  -ngl 99

For macOS/Metal builds, drop the -DGGML_CUDA=ON CMake flag; the build system will target Metal for Apple Silicon automatically. An MLX companion (Ternary-Bonsai-27B-mlx-2bit) is also published for native Apple Silicon inference outside of llama.cpp. The entire stack is released under an Apache 2.0 license — no royalties, no non-commercial restrictions, immediately usable in production software.

DSpark Speculative Decoding: The Throughput Multiplier

Speculative decoding accelerates autoregressive inference by using a small "drafter" model to propose several tokens at once; the target model then verifies them in a single parallel forward pass, accepting or rejecting each. Accepted drafts deliver free throughput because the verification step is parallel and far cheaper than serial generation.

The DSpark drafter bundled with the prism ternary model is a purpose-built six-layer, block-parallel transformer that taps hidden states from five evenly spaced layers of the target Bonsai model. Its embedding and output head are shared with the target model, so the drafter adds only roughly 0.5 GB of unique weights at serving precision (the Q4_1 file totals 1.95 GB, but most of that is shared state). Training used the DSpark recipe: a diffusion-flavored block-denoising objective, survival-probability-weighted distillation, and per-source-normalized hidden-state taps, with draft block size selected from a measured verify-cost model of the serving stack.

The losslessness guarantee matters for production deployments: the verification step preserves the target model's output distribution exactly. Accepted tokens are statistically indistinguishable from tokens the full ternary model would have generated without any drafter present. The 1.34× throughput gain on H100 (98.0 → 131.8 tok/s, with an accepted draft length of about 3.7 tokens at draft depth 4) is therefore quality-neutral — a speed improvement, not a speed/quality trade-off.

Key Takeaways for Developers

• 94.6% of FP16 quality at 1.71 bits per weight: The prism ternary bonsai 27b gguf averages 80.49 across 15 benchmarks versus 85.07 for FP16 — the strongest quality-per-bit ratio documented in this weight class at sub-2-bit depth.
• ~5.9 GB weight footprint with 262K-token context: A model class that previously required 50+ GB of VRAM now fits on a single consumer GPU or a base-tier MacBook Pro (resident footprint about 7.2 GB at inference), with full long-context capability intact.
• Native ternary training, not PTQ: Because ternary weights are a training objective rather than a post-hoc approximation, the model outperforms PTQ 2-bit equivalents by ~8 benchmark points while using less memory — the two advantages are directly connected.
• Laptop-viable at practical speeds: 26.2 tok/s on Apple M5 Pro exceeds comfortable human reading speed; 44.0 tok/s on M5 Max suits interactive coding assistants and agent loops.
• Modular file structure: The DSpark drafter, vision tower, and group-64 alternate pack are independent add-ons — include only what your application actually needs.
• Apache 2.0 license: No royalties, no non-commercial clauses; immediately deployable in commercial products.
• Prism ML's llama.cpp fork is required: The custom ternary kernels are not yet merged into upstream llama.cpp — factor this dependency into integration planning and CI pipelines.
• Active community traction: The GGUF repository has already gathered 652 likes on Hugging Face and is circulating within the 2-bit and GGUF communities — a signal of real-world evaluation, though you should still benchmark it on your own workload.

What Comes Next for the Prism Ternary Approach and Sub-2-Bit Models

The prism ternary bonsai release is best understood as a proof point in a broader architectural shift. The dominant open-weight efficiency strategy for the past several years has been post-training quantization: take a finished FP16 model and compress it. Results are strong down to about 4 bits, acceptable at 3 bits, and noticeably degraded below that. Ternary and binary training sidestep this ceiling by making low-bit representation part of the learning objective itself — the model is never "degraded" from a higher-precision state. The trade-off is a lower raw representational ceiling during training, exchanged for a model that is simultaneously smaller, faster, and more accurate at its deployed bit-width than any PTQ alternative can achieve.

Community traction on Hugging Face suggests developers are actively evaluating this approach rather than merely watching from the sidelines. The practical trajectory is clear: as hardware vendors keep optimizing silicon for narrow-integer matrix operations — Apple's AMX units, NVIDIA's upcoming architectures — natively ternary models will see further inference acceleration without any software changes. The Bonsai 1-bit companion targeting phone-class hardware points toward a near-term future where on-device reasoning at smartphone scale is routine rather than experimental.

As these techniques mature, the defining metric for model efficiency will shift from parameter count toward quality per byte — and on that metric, models like the prism ternary bonsai are already rewriting expectations. For developers evaluating local deployment today, the implication is concrete: workloads that appeared to require cloud API calls or dedicated GPU infrastructure may now be viable candidates for private, zero-latency, on-device deployment. The practical starting point is simple — download Ternary-Bonsai-27B-Q2_0.gguf from Hugging Face, build Prism ML's llama.cpp fork, and benchmark it against your own workload before drawing conclusions from aggregate scores alone.