If you’re shopping for a mini PC and want to run large language models locally, you might be interested in the Bonsai 27B model from PrismML. This model is based on Qwen3.6 27B but is compressed to fit on an 8GB graphics card, making it accessible for users with modest hardware.
The Bonsai 27B model is compressed using a technique that stores each weight as either a +1 or -1 with one shared scale value for each group of 128 weights, resulting in about 1.125 bits per weight. This compression allows the model to fit into just under 4GB of VRAM, making it feasible to run on an 8GB GPU. There are two versions: a 1-bit version at 4.4GB and a ternary version that adds zero as a third possible value.
In terms of performance, Bonsai 27B excels in math and logic tasks, scoring 91.7 on math benchmarks compared to 95.3 for the full Qwen model. It also handles document summarization and instruction following well, making it useful for a variety of tasks. However, it does have some weaknesses, particularly in tool calling and image processing, where it scores lower than Qwen and Gemma models.
For those interested in running local AI models on a mini PC, the Bonsai 27B model offers a compelling option. It runs smoothly on an 8GB GPU, making it accessible for users who might not have the latest high-end hardware. The model’s strengths in math, logic, and document summarization make it a versatile tool for various tasks. However, if you need strong tool calling or image processing capabilities, you might still prefer Qwen or Gemma models.
For more details, you can check out the original article here.
