Machine Learning Developer Advocate | LLMs, AI Agents & RAG | Shipping Open Source AI Apps | X (65K+)
Microsoft just changed the game! 🔥 They've open-sourced bitnet.cpp: a blazing-fast 1-bit LLM inference framework that runs directly on CPUs. You can now run 100B parameter models on local x86 CPU devices with up to 6x speed improvements and 82% less energy consumption. They also released BitNet b1.58 2B4T, the first functional open-source model that uses just 1.58 bits for weights instead of the usual 16 or 32 bits. BitNet b1.58 uses ternary weights (-1, 0, +1) and 8-bit activations, reducing memory needs by ~6x compared to similar-sized models while matching their performance on benchmarks Link to the paper and the repo in the comments! If you're interested in ML, LLMs, RAG, and AI Agents and want to receive Apps and tutorials every week, subscribe to AI Engineering (for free): https://lnkd.in/d9REmcqK
That's interesting. Can we already load it with Ollama?
Thanks for sharing, Sumanth, that's pretty fast
The concept is nice, but i heared (not tested myself) that the performance degrades so much that it’s almost unusable. If there is anyone who had a positive experience? I would love to know more!
Great breakdown An article i wrote to set this up https://medium.com/stuvalley-ai-nexus/cpu-based-llm-for-gpu-poor-115-72-909d9c9e6539
Thanks for sharing, Sumanth
Ternary weights? 🤯 My brain is still processing this. Huge for the future of AI. Thanks for the link 🔥
Nice share. Open source small LLM are the future. Just this week I experiment a lot with different models with Ollama.
Large models getting smaller and I’m here for it, definitely checking this out!
Can this be considered a replacement for ollama?........is there a way to run models provided by ollama, using this?
Machine Learning Developer Advocate | LLMs, AI Agents & RAG | Shipping Open Source AI Apps | X (65K+)
1wLink to the Paper: https://arxiv.org/abs/2504.12285 GitHub: https://github.com/microsoft/BitNet