Microsoft BitNet: inference framework for 1-bit LLMs | Hacker News Reader

Microsoft BitNet: inference framework for 1-bit LLMs

github.com

173 pointsgaleos33 comments

I'm enthusiastic about BitNet and the potential of low-bit LLMs - the papers show impressive perplexity scores matching full-precision models while drastically reducing compute and memory requirements. What's puzzling is we're not seeing any major providers announce plans to leverage this for their flagship models, despite the clear efficiency gains that could theoretically enable much larger architectures. I suspect there might be some hidden engineering challenges around specialized hardware requirements or training stability that aren't fully captured in the academic results, but would love insights from anyone closer to production deployment of these techniques.

I find it a little confusing as well. I wonder if its because so many of these companies have went all in on the "traditional" approach that deviating now seems like a big shift?

I think that since training must happen on a non-bitnet architecture, tuning towards bitnet is always a downgrade on it's capabilities, so they're not really interested in it. But maybe they could be if they'd offer cheaper plans, since it's efficiency is relatively good.

I think the real market for this is for local inference.

I suppose hardware support would be very helpful, new instructions for bitpacked operations?

People are almost certainly working on it. The people who are actually serious and think about things like this are less likely to just spout out "WE ARE BUILDING A CHIP OPTIMIZED FOR 1-BIT" or "WE ARE TRAINING A MODEL USING 1-BIT" etc, before actually being quite sure they can make it work at the required scale. It's still pretty researchy.

For anyone that hasn't read the previous papers before the "1.58-bit" part comes from using 3 values (-1, 0, 1) and log2[3]=1.58...

Has some one made an FPGA or ASIC implementation yet? It feels like it should be easy (and people would snap up for inference).

Sorry for a stupid question but to clarify, even though it is a 1-bit model, it is supposed to be working with any types of embeddings, even taken from larger LLMs(in their example, they use HF1BitLLM/Llama3-8B-1.58-100B-tokens). I.e. it doesn't have an embedding layer built-in and relies on embedding provided separately?

No. You can't put any type of embedding in.

Can anyone help me understand how this works without special bitnet precision-specific hardware? Is special hardware unnecessary? Maybe it just doesn't reach the full bitnet potential without it? Or maybe it does, with some fancy tricks? Thanks!

I haven't checked this one out yet, but a common trick is using combinations of instructions and data invariants allowing you to work in "lanes".

The easiest example is xor, which can trivially be interpreted as either xoring one large integer or xoring a vector of smaller integers.

Take a look at the SWAR example here [0] as a pretty common/easy example of that technique being good for something in the real world.

Dedicated hardware is almost always better, but you can still get major improvements with a little elbow grease.

[0] https://nimrod.blog/posts/algorithms-behind-popcount/

This is extremely easy to implement in-FPGA.

While fancy hardware would make it faster, what you are comparing it to is a bunch of floating point and large number multiplication. I believe in this case they just use a look up table:

If one value is 0, it is 0.

If the signs are different, it is -1.

If the signs are the same, it is 1.

I’m sure those can be done with relatively few instructions using far less power hungry hardware.

The major benefit would be its significant decrease in memory consumption, rather than the compute itself. The major bottleneck of the current LLM infra is typically memory bandwidth and that's the reason why those chip industries are going crazy on HBM. Surely compute optimization helps but this is useful even without any hardware changes.

Inference speeds go brrrr as well.

I'm glad Microsoft uses Bash in the example, instead of their own Windows shells. As a user I would like having something like "Git Bash" for Windows built in the system, as default shell.

WSL is where it's at today. It's not quite what you're asking for, as it is a separate virtual OS, but the integration is so tight that it feels like you're using your favorite shell natively in Windows.

> integration is so tight that it feels like you're using your favorite shell natively in Windows

WSL1 certainly felt that way, WSL2 just feels like any other virtualization manager and basically works the same. Not sure why people sings the praise of WSL2, I gave it a serious try for months but there is a seemingly endless list of compatibility issues which I never had with VMWare or VirtualBox, so I just went back to those instead and the experience is the same more or less.

Probably because it has relatively painless GPU sharing with pass through. As far as I know that sort of feature requires a hypervisor-level VM, which is not something you get with VirtualBox.

Someone correct me if I'm wrong, but I think you can use a KVM or QEMU backend for VirtualBox and that way get GPU pass-through. Probably not out of the box though.

Close Windows, many doors open

The WSL2 GPU passthrough is more like a virtual GPU than KVM style device passthrough. I believe it's effectively a device specific linux userland driver to device specific windows kernel driver with a linux kernel shim bridging the too. If I recall correctly, the linux userland drivers are actually provided by the windows driver.

Just install Cygwin.

Not sure what you mean by “default shell”. The default shell on Windows is this: https://en.wikipedia.org/wiki/Windows_shell. I don’t suppose you mean booting into Bash. Windows doesn’t have any other notion of a default shell.

I used Cygwin for more than a decade. I prefer Git Bash (msys-based).

I thought msysgit was also cygein based

They are using Windows shells, just not on Macs. This is what the caption for the video says:

> A demo of bitnet.cpp running a BitNet b1.58 3B model on Apple M2:

And this is in their Windows build instructions:

> Important! If you are using Windows, please remember to always use a Developer Command Prompt / PowerShell for VS2022 for the following commands

Neat. Would anyone know where the SDPA kernel equivalent is? I poked around the repo, but only saw some form of quantization code with vectorized intrinsics.

I assume it is not as powerful at some tasks than full sized model, so what would one use this model for ?

When will AIs learn to no bla bla bla by default

No GPU inference support?

> that support fast and lossless inference of 1.58-bit models on CPU (with NPU and GPU support coming next).

1.58bpw is not “1 bit”.

-1,1,0 => log2(3) = 1.58

Reply on news.ycombinator.com