AI engineers claim new algorithm reduces AI power consumption by 95%
tomshardware.com
tomshardware.com
ABSTRACT
Large neural networks spend most computation on floating point tensor multiplications. In this work, we find that a floating point multiplier can be approximated by one integer adder with high precision. We propose the linear-complexity multiplication (L-Mul) algorithm that approximates floating point number multiplication with integer addition operations. The new algorithm costs significantly less computation resource than 8-bit floating point multiplication but achieves higher precision. Compared to 8-bit floating point multiplications, the proposed method achieves higher precision but consumes significantly less bit-level computation. Since multiplying floating point numbers requires substantially higher energy compared to integer addition operations, applying the L-Mul operation in tensor processing hardware can potentially reduce 95% energy cost by elementwise floating point tensor multiplications and 80% energy cost of dot products. We calculated the theoretical error expectation of L-Mul, and evaluated the algorithm on a wide range of textual, visual, and symbolic tasks, including natural language understanding, structural reasoning, mathematics, and commonsense question answering. Our numerical analysis experiments agree with the theoretical error estimation, which indicates that L-Mul with 4-bit mantissa achieves comparable precision as float8 e4m3 multiplications, and L-Mul with 3-bit mantissa outperforms float8 e5m2. Evaluation results on popular benchmarks show that directly applying L-Mul to the attention mechanism is almost lossless. We further show that replacing all floating point multiplications with 3-bit mantissa L-Mul in a transformer model achieves equivalent precision as using float8 e4m3 as accumulation precision in both fine-tuning and inference.
Presumably there will be a lot of interest.
You still need the GPU parallelism though.
Once the "math is done" quite likely it would have paid off better than most investments for the top people to have spent a few short years working with grossly underpowered hardware until they could come up with amazing results there before scaling up. Rather than grossly overpowered hardware before there was even deep understanding of the underlying processes.
When you think about it, what we have seen from the latest ultra-high-powered "thinking" machines is truly so impressive. But if you are trying to fool somebody into believing that it's a real person it's still not "quite" there.
Maybe a good benchmark would be to take a regular PC, and without reliance on AI just pull out all the stops and put all the effort into fakery itself. No holds barred, any trick you can think of. See what the electronics is capable of this way. There are some smart engineers, this would only take a few years but looks like it would have been a lot more affordable.
Then with the same hardware if an AI alternative is not as convincing, something has got to be wrong.
It's good to find out this type of thing before you go overboard.
Regardless of speed or power, I never could have gotten an 8-bit computer to match the output of a 32-bit floating-point algorithm by using floating-point myself. Integers all the way and place the decimal where it's supposed to be when you're done.
Once it's really figured out, how do you think it would feel being the one paying the electric bills up until now?
Of course, the impact is that AI will continue to become cheaper to use, and induced demand will continue the feedback loop driving the market as a result.
Just summary features: save me 20min of reading a transcript, turn it into 20s. That's a huge enabler.
Effective and information-dense communication is really hard. That doesn't mean we should just accept the useless fluff surrounding the actual information and/or analysis. People could learn a lot from the Ignoble Prize ceremony's 24/7 presentation model.
Sadly, it seems we are heading towards a future where you may need an LLM to distill the relevant information out of a sea of noise.
Not all content is worth consuming, and not all content is dense.
But if that's the only place that contained the information you needed, then you have no choice.
There's a lot of material out there that is badly written, badly organized, badly presented. LLM's can be a godsend for extracting the information you actually need without wasting 20 minutes wading through the muck.
If I paste the actual non-trivial code, it starts deviating fast. And it isn’t too complex, it’s just less like “parallel sort two arrays” and more like “wait for an image on a screenshot by execing scrot (with no sound) repeatedly and passing the result to this detect-cv2.py script and use all matching options described in this ts type, get stdout json as in this ts type, and if there’s a match, wait for the specified anim timeout and test again to get the settled match coords after an animation finishes; throw after a total timeout”. Not a rocket science, pretty dumb shit, but right there they fall flat and start imagining things, heavily.
I guess it shines if you ask it to make an html form, but I couldn’t call that life-changing unless I had to make these damn forms all day.
I don't fear that LLMs are going to take my job as a developer. I'm pretty sure they mark a further decrease in the quality and coherence of software, along with a rapid increase in the quantity of code out there, and that seems likely to provide me with reliable employment forever. I'm basically employed in fixing bugs that didn't need to exist in the first place and that seems to cover a lot of software dev.
I serve Cody a direct question about 1-3 times a week. Of that, it gets maybe 50% correct on the first try. I don’t bother with a second try because by then I’ve already spent the equivalent amount of time looking at the relevant library source code and/or docs would have taken.
Even allowing for some hyperbole, your programming experience is extremely different from mine. Looking anything up outside the IDE, let alone via Google, is by far the exception for me rather than the rule.
I've long suspected that this kind of difference explains a lot of the difference in how Copilot is perceived.
Maybe you don't push your boundaries as an engineer and thus rarely need to know new things or at least learn new API surfaces. Maybe you don't know how to effectively prompt an LLM. Maybe you lack the mastery to analyze and refine the results. Maybe you just like doing things the slow way. I too remember a time as an early programmer where I eschewed even Intellisense and basic auto complete...
I'd recommend learning a bit more and practicing some humility and curiosity before condemning an entire class of engineers just because you don't understand their workflow. Just because you've had subpar experiences with a new tool doesn't mean it's not a useful tool in another engineer's toolkit.
Evaluating my skills based on how I evaluated someone else's skills when they tell me about their abilities with and without a crutch, and throwing big academic sounding expressions with 'effect' in them might be intimidating to some but to me it just transparently sounds pretentious and way off mark, since, like I said, you have zero data about my abilities or output.
> I'd recommend learning a bit more and practicing some humility and curiosity before condemning an entire class of engineers
You're clearly coming from an emotional place because you feel slighted. There is no 'class of engineers' in my evaluation. I recommend reading comments more closely, thinking about their content, and not getting offended when someone points out signs of lacking skills, because you might just be advertising your own limitations.
Didn't you just do that to an entire class of engineers:
> Claiming LLMs are a massive boost for coding productivity is becoming a red flag that the claimant has a tenuous grasp on the skills necessary
Anyway,
> Evaluating my skills based on how I evaluated someone else's skills when they tell me about their abilities with and without a crutch
Your argument rests on the assumption that LLMs are a "crutch", and you're going to have to prove that before the rest of your argument holds any water.
It sucks getting generalized, doesn't it? Feels ostracizing? That's the exact experience someone who productively and effectively uses LLMs will have upon encountering your premature judgement.
> You're clearly coming from an emotional place because you feel slighted.
You start off your post upset that I'm "making claims" about your skills (I used the word "maybe" intentionally, multiple times), and then turn around and make a pretty intense claim about me. I'm not "clearly" coming from an emotional place, you did not "trigger" me, I took a moment to educate you about being overly judgemental before fully understanding something, and pointed out the inherent hypocrisy.
> you might just be advertising your own limitations
But apparently my approach was ineffective, and you are still perceiving a world where people who approach their work differently than you are inferior. Your toxic attitude is unproductive, and while you're busy imagining yourself as some masterful engineer, people are out there getting massive productivity boosts with careful application of cutting-edge generative technologies. LLMs have been nothing short of transcendental to a curious but skilled mind.
Not really. He said "if you claim LLM's are next thing since sliced butter I am doubting your abilities". Which is fair. It's not really a class as much as a group.
I've never been wowed over by LLMs. At best they are boilerplate enhancers. At worst they write plausibly looking bullshit that compiles but breaks everything. Give it something truly novel and/or fringe and it will fold like a deck of cards.
Even latest research called LLM's benefits into question: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4945566
That said. They are fine at generating commit messages and docs than me.
No, OP said:
> Claiming LLMs are a massive boost for coding productivity is becoming a red flag that the claimant has a tenuous grasp on the skills necessary
Quotation marks are usually reserved for direct quotes, not paraphrases or straw mans.
> I've never been wowed over by LLMs.
Cool. I have, and many others have. I'm unsure why your experience justifies invalidating the experiences of others or supporting prejudice against people who have made good use of them.
> Give it something truly novel and/or fringe and it will fold like a deck of cards.
Few thoughts are truly novel, most are derivative or synergistic. Cutting edge LLMs, when paired with a capable human, are absolutely capable of productive work. I have long, highly technical and cross-cutting discussions with GPT 4o which I simply could not have with any human that I know. Humans like that exist, but I don't know them and so I'm making due with a very good approximation.
Your and OP's lack of imagination at the capabilities of LLMs are more telling than you realize to those intimate with them, which is what makes this all quite ironic given that it started from OP making claims about how people who say LLMs massively boost productivity are giving tells that they're not skilled enough.
Not on HN. Customary is to use > paragraph quotes like you did. However I will keep that in mind.
> Cool. I have, and many others have. I'm unsure why your experience justifies invalidating the experiences of others
If we're both grading a single student (LLM) in same field (programming), and you find it great and I find it disappointing, it means one of us is scoring it wrong.
I gave papers that demonstrate its failings, where is your counter-proof?
> Your and OP's lack of imagination at the capabilities of LLMs
It's not lack of imagination. It's terribleness of results. It can't consistently write good doc comments. I does not understand the code nor it's purpose, but roughly guesses the shape. Which is fine for writing something that's not as formal as code.
It can't read and understand specifications, and even generate something as simple as useful API for it. The novel part doesn't have to be that novel just something out of its learned corpus.
Like Yaml parser in Rust. Maybe Zig or something beyond it's gobbled data repo.
> Few thoughts are truly novel, most are derivative or synergistic.
Sure but you still need A mind to derive/synergize the noise of everyday environment into something novel.
It can't even do that but remix data into plausibly looking forms. A stochastic parrot. Great for DnD campaign. Shit for code.
Hacker News is not some strange place where the normal rules of discourse don't apply. I assume you are familiar with the function of quotation marks.
> If we're both grading a single student (LLM) in same field (programming), and you find it great and I find it disappointing, it means one of us is scoring it wrong.
No, it means we have different criteria and general capability for evaluating the LLM. There are plenty of standard criteria which LLMs are pitted against, and we have seen continued improvement since their inception.
> It can't consistently write good doc comments. I does not understand the code nor it's purpose, but roughly guesses the shape.
Writing good documentation is certainly a challenging task. Experience has led me to understand where current LLMs typically do and don't succeed with writing tests and documentation. Generally, the more organized and straightforward the code, the better. The smaller each module is, the higher the likelihood of a good first pass. And then you can fix deficiencies in a second, manual pass. If done right, it's generally faster than not making use of LLMs for typical workflows. Accuracy also goes down for more niche subject material. All tools have limitations, and understanding them is crucial to using them effectively.
> It can't read and understand specifications, and even generate something as simple as useful API for it.
Actually, I do this all the time and it works great. Keep practicing!
In general, the stochastic parrot argument is oft-repeated but fails to recognize the general capabilities of machine learning. We're not talking about basic Markov chains, here. There are literally academic benchmarks against which transformers have blown away all initial expectations, and they continue to incrementally improve. Getting caught up criticizing the crudeness of a new, revolutionary tool is definitely my idea of unimaginative.
Language is all about context. I wasn't trying to be deceitful. And on HN I've never seen anyone using quotation marks to quote people.
> Writing good documentation is certainly a challenging task.
Doctests isn't same as writing documentation. Doctest are the simplest form of documentation. Given function named so and so write API doc + example. It could not even write example that passed syntax check.
> Actually, I do this all the time and it works great. Keep practicing!
Then you haven't given it interesting/complex enough problems.
Also this isn't about practice. It's about its capabilities.
> In general, the stochastic parrot argument is oft-repeated but fails to recognize the general capabilities of machine learning.
I gave it write YAML parser given Yaml org spec, and it wrote following struct:
enum Yaml {
Scalar(String),
List(Vec<Box<Yaml>>),
Map(HashMap<String, Box<Yaml>>),
}
Whatever LLM's are they aren't intelligent. Or they have attention spans of a fruit fly and can't figure out basic differences.
It's still unclear how this apparent lack of knowledge of basic writing mechanics would justify your use of quotation marks to attempt a straw man argument wherein you deliberately attempted to convince me that OP said something completely different.
> Doctests isn't same as writing documentation. Doctest are the simplest form of documentation. Given function named so and so write API doc + example. It could not even write example that passed syntax check.
That truly sounds like a skill issue. This no-true-Scotsman angle is silly. I said documentation and tests, I don't know how you got "doctests" out of that. I said "documentation", and "tests". I didn't say "the simplest form of documentation", that is another straw man on your behalf.
> Then you haven't given it interesting/complex enough problems.
Wow, the arrogance. There is absolutely nothing to justify this assumption. It's exceedingly likely that you yourself aren't capable of interacting meaningfully with LLMs for one reason or another, not that I haven't considered interesting or complex problems. I bring some extraordinarily difficult cross-domain problems to these tools and end up satisfied with the results far more often than not.
My argument is literally that cutting-edge LLMs excel with complex problems, and they do in many cases in the right hands. It's unfortunate if you can't find these problems "interesting" enough, but that hasn't stopped me from getting good enough results to justify using an LLM during research and development.
> Also this isn't about practice. It's about its capabilities.
Unfortunately, this discourse has made it clear that you do need considerable practice, because you seem to get bad results, and you're more interested in defending those bad results even if it means insulting others, instead of just considering that you might not quite be skilled enough.
> This is the stochastic parrot in action. Why? Because it tried to pass of JSON like structure as YAML.
That proves its stochasticity, but it doesn't prove it is a "stochastic parrot". As long as you lack the capability to realistically assess these models, it's no wonder that you've had such bad experiences. You didn't even bother clarifying which LLM you used, nor did you mention any parameters of your experiment or even if you attempted multiple trials with different LLMs or prompts. You failed to follow the scientific method and so it's no surprise that you got subpar results.
> Whatever LLM's are they aren't intelligent.
You have demonstrated throughout this discussion that you aren't capable of assessing machine intelligence. If you learned how to be more open-minded and took the time to learn more about these new technologies, instead of complaining about contemporary shortcomings and bashing those who do benefit from the technologies, it would likely open many doors for you.
What are you on about? Doctest is the simplest form of documentation and test. I.e. you don't have to write an in-depth test, you just need to understand what the function does. I expect even juniors can write a doctest that passes the compiler check. Not a good, not a passing one doctest, a COMPILING one. It's rate of writing a passing one was even worse.
> Wow, the arrogance. There is absolutely nothing to justify this assumption.
Ok, then. Prove what exactly hard problems did you give it?
I gave my examples, I noticed it fails at complex tasks like YAML parser in an unknown language.
I noticed when confronted with anything harder than writing pure boilerplate, it fails. E.g. it would fail 10% of the time.
> Unfortunately, this discourse has made it clear that you do need considerable practice
You can practice with a stochastic parrot all you want, it won't make it an Einstein. Programming is all about converting requirements to math, and LLMs aren't good at it. Do I need to link stuff like doing basic calculation and counting 'r' in the word 'strawberries'.
The best you can do is half the error rate, but that follows a power law. You need to double the energy to half the error rate. So unless you intend to boil the surface of the Earth to get it to be decent at programming, I don't think it's going to change anytime soon.
> You have demonstrated throughout this discussion that you aren't capable of assessing machine intelligence.
Pure ad hominem. You've demonstrated nothing outside your ""Trust me bro, it's not a bubble"" and ""You're wrong"". I'm using double double quotes so you don't assume I'm quoting you.
> You didn't even bother clarifying which LLM you used.
For YAML parser, I used Chat GPT-4o at my friend's place. For the rest of the tasks I used JetBrains AI assistant, which is a mix of Chat GPT-4, GPT-4o and GPT-3.
I’m not sure when and why reading documentation and man pages became a sign of a lack of skill. Watch a presentation by someone like Brian Kernighan and you’ll see him joke about looking up certain compiler flags for the thousandth time!
Personally I work in C, C#, F#, Java, Kotlin, Swift, R, Ruby, Python, Postgres SQL, MySQL SQL, TypeScript, node, and whatever hundreds of libraries and DSLs are built on top. Yes, I have to look up documentation and with regularity.
What I'm not good at is memorizing API's and libraries that all use different verbs and nouns for the same thing, and other such things that are immaterial to the actual work. How do you use a mutation observer again? Hell if I remember the syntax but I know the concept, and copilot will probably spit out what I want, and I'll easily verify the output. Or how do you copy an array in JS? Or print a stack trace? Or do a node walk? You can either wade through google and stackoverflow, or copilot can tell you instantly. And I can very quickly tell if the code copilot gave me is sensible or not.
On another note, even if you are experienced it helps when doing new stuff and you don’t know the proper syntax for what you want. For example let’s say your using flutter, you can just type
// bold
And it will help put the proper bold stuff in there.
I'm firmly in #2. My other comment goes over how.
I'm intrigued to see how devs in #1 grow. One might be wary those devs would grow into bad habits and not thinking for themselves, but it might be a case of the ancient Greek rant against written books hindering memorization. Could be that they'll actually grow to be even better devs unburdened by time wasted on trivial details.
I never thought it would be ideal if it was otherwise, so I guess so.
When I first considered neural nets from state-of-the art vendors to assist with some non-linguistic situations over 30 years ago, it wasn't quite ready for prime time and I could accept that.
I just don't have generic situations all the time which would benefit me, so it's clearly my problems that have the deficiencies ;\
What's being done now with all the resources being thrown at it is highly impressive, and gaining all the time, no doubt about it. It's nice to know there are people that can afford it.
I truly look forward to more progress, and this may be the previously unreached milestone I have been detecting that might be a big one.
Still not good enough for what I need yet so far though. And I can accept that as easily as ever.
That's why I put up my estimation that not all of those 30+ years has been spent without agonizing over something ;)
That is a good correlation when you think about how much more energy-efficient transistors are than vacuum tubes.
Vacuum tube computers were a thing for a while, but it was more out of desperation than systematic intellectual progress.
OTOH you could look at the present accomplishments like it was throwing more vacuum tubes at a problem that can not be adequately addressed that way.
What turned out to be a solid-state solution was a completely different approach from the ground up.
To the extent a more power-saving technique using the same hardware is only a matter of different software approaches, that would be something that realistically could have been accomplished before so much energy was expended.
Even though I've always thought application-specific circuits would be what really helps ML and AI a lot, and that would end up not being the exact same hardware at all.
If power is truly being wasted enough to start rearing its ugly head, somebody should be able to figure out how to fix it before it gets out-of-hand.
Ironically enough with my experience using vacuum tubes, I've felt that there were some serious losses in technology when the research momentum involved was so rapidly abandoned in favor of "solid-state everything" at any cost.
Maybe it is a good idea to abandon the energy-intensive approaches, as soon as anything completely different that's the least bit promising can barely be seen by a gifted visionary to have a glimmer of potential.
The momentum with LLMs and allied technology may last till it keeps on improving even by a few percentage points and keeps shattering human created new benchmarks every few months
Its iteritive, there are plenty of cul-de-sacs and failures. You can't really optimise until you have something that works and its a messy process that is inefficient.
You're looking at this with hindsight.
Even with PCIe 5.0 and 16 lanes, you only get 64 GB/s of bandwidth. If you're trying to run a model too big for your GPU, then for every token, it has to reload the entire model. With a 70B parameter model, 8 bit quantization, you're looking at just under 1 token/sec just from having to transfer parts of the model in constantly. Making the actual computation faster won't make it any faster.
Your best bet would likely be a laptop that has integrated system RAM with VRAM, but I don't think any of those offer enough RAM to store an entire 70B model. A 7B parameter model would work fine, but you could do those on a consumer-grade GPU anyways.
If you're training models with billions of parameters, you're still gonna need that.
Here https://news.ycombinator.com/item?id=41784591 but even before that. It is possibly one of those obvious ideas to people steeped in this.
To me intuitively using floats to make ultimatelty boolean like decisions seems wasteful but that seemed like the way it had to be to have diffetentiable algorithms.
I tried implementing this for AVX512 with tinyBLAS in llamafile.
inline __m512 lmul512(__m512 x, __m512 y) {
__m512i sign_mask = _mm512_set1_epi32(0x80000000);
__m512i exp_mask = _mm512_set1_epi32(0x7F800000);
__m512i mant_mask = _mm512_set1_epi32(0x007FFFFF);
__m512i exp_bias = _mm512_set1_epi32(127);
__m512i x_bits = _mm512_castps_si512(x);
__m512i y_bits = _mm512_castps_si512(y);
__m512i sign_x = _mm512_and_si512(x_bits, sign_mask);
__m512i sign_y = _mm512_and_si512(y_bits, sign_mask);
__m512i exp_x = _mm512_srli_epi32(_mm512_and_si512(x_bits, exp_mask), 23);
__m512i exp_y = _mm512_srli_epi32(_mm512_and_si512(y_bits, exp_mask), 23);
__m512i mant_x = _mm512_and_si512(x_bits, mant_mask);
__m512i mant_y = _mm512_and_si512(y_bits, mant_mask);
__m512i sign_result = _mm512_xor_si512(sign_x, sign_y);
__m512i exp_result = _mm512_sub_epi32(_mm512_add_epi32(exp_x, exp_y), exp_bias);
__m512i mant_result = _mm512_srli_epi32(_mm512_add_epi32(mant_x, mant_y), 1);
__m512i result_bits = _mm512_or_si512(
_mm512_or_si512(sign_result, _mm512_slli_epi32(exp_result, 23)), mant_result);
return _mm512_castsi512_ps(result_bits);
}
Another example, if we have for example 1.9 * 1.9 then we need to account for overflow in (0.9 + 0.9) and this seems to induce similar overhead as expressing numbers as (1-0.05)*2 .
If I could have a SWAG at it I would say a low resolution model like llama-2 would probably be just fine (llama-2 quantizes without too much headache) but a higher resolution model like llama-3 probably not so much, not without massive retraining anyways.
Also,
> Additionally, it reduces energy consumption by 55.4% to 70.0%
With humility, I don't know what that means. It seems like some dubious math with percentages.
I would start by downloading a 1.58 model such as: https://huggingface.co/HF1BitLLM/Llama3-8B-1.58-100B-tokens
Run the non-quantized version of the model on your 3090/4090 gpu and observe the power draw. Then load the 1.58 model and observe the power usage. Sure, the numbers have a wide range because there are many gpu/npu to make the comparison.
That said, time to market has been more important than any cares of efficiency for some time. Now and in the future, there is more of a focus on it as the expenses in equipment and power have really grown.
The 95% gains is specifically only for multiplication operations, inference is compute light and memory heavy in the first place so the actual gains would be far less smaller .
Tech journalism (all journalism really) can hardly be trusted to publish grounded news with the focus on clicks and revenue they need to survive.
We have a winner. Glad that came from someone not in my lectures on ML network design
Honestly, thanks for beeting me to this comment
Right now the only way to gain real knowledge is actually to read comments of those articles.
I'm not claiming it's not possible, nor am I claiming that it's not true, or, at least, honest.
But, there will need to be evidence that using real machines, and using real energy an _equivalent performance_ is achievable. A defense that "there are no suitable chips" is a bit disingenuous. If the 95% savings actually has legs some smart chip manufacturer will do the math and make the chips. If it's correct, that chip making firm will make a fortune. If it's not, they won't.
Terrible logic. By a similar logic we wouldn't be using python for machine learning at all, for example (or x86 for compute). Yet here we are.
They are and have been buying AMD CPUs for a while now, which says something about AMD and Intel.
I like that the narrative has changed from "AI only runs on Cuda" to "sure it runs fine on AMD if you must"
> In this section, we show that L-Mul is more precise than fp8 e4m3 multiplications
> To be concise, we do not consider the rounding to nearest even mode in both error analysis and complexity estimation for both Mul and L-Mul
These two statements together are non-sensical. Sure, if you analyze accuracy while ignoring the part of the algorithm that gives you accuracy in the baseline you can derive whatever cherry-picked result you want.
The multiplication of two floating point values if you round to nearest even will be the correctly rounded result of multiplying the original values at infinite precision, this is how floating point rounding usually works and what IEEE 754 mandates for fundamental operations if you choose to follow those guidelines (e.g., multiplication here). But not rounding to nearest even will result in a lot more quantization noise, and biased noise at that too.
> applying the L-Mul operation in tensor processing hardware can potentially reduce 95% energy cost by elementwise floating point tensor multiplications and 80% energy cost of dot products
A good chunk of the energy cost is simply moving data between memories (especially external DRAM/HBM/whatever) and along wires, buffering values in SRAMs and flip-flops and the like. Combinational logic cost is usually not a big deal. While having a ton of fixed-function matrix multipliers does raise the cost of combinational logic quite a bit, at most what they have will probably cut the power of an overall accelerator by 10-20% or so.
> In this section, we demonstrate that L-Mul can replace tensor multiplications in the attention mechanism without any loss of performance, whereas using fp8 multiplications for the same purpose degrades inference accuracy
I may have missed it in the paper, but they have provided no details on (re)scaling and/or using higher precision accumulation for intermediate results as one would experience on an H100 for instance. Without this information, I don't trust these evaluation results either.
So it turns the (1+a)(1+b) into 1+a+b. Which is definitely not the same! But it turns out, machine guessing apparently doesn't care much about the difference.
Am I missing something?
Feels like multiplication shouldn't be needed for convergence, just monotonicity? I wonder how well it would perform if the model was actually trained the same way.
You're going to have tolerance on the result anyway, so what's a little more error. :)
https://news.ycombinator.com/item?id=41816598
This has been done for decades in digital circuits, FPGA’s, Digital Signal Processing, etc. Floating point is both resource and power intensive and using FP without the use of dedicated FP processing hardware is something that has been avoided and done without for decades unless absolutely necessary.
At basic level it is very simple: A 10 bit bus gives you the ability to represent numbers between 0 and 1 with a resolution of approximately 0.001. 12 bits would be four times better. Integer circuits can do the math in one clock cycle. Hardware multipliers do the same. To rescale the numbers after multiplication you just take the N high bits, where N is your bus width; which is a zero clock-cycle operation. Etc.
In training a neural network, the back propagation math can be implemented using almost the same logic used for a polyphase FIR filter.
Their rediscovery of fixed point was bad enough but the “omg if we represent poses as quaternions everything works better” makes any game engine dev for the last 30 years explode.
Why not publish some actual benchmarks that prove your claim in even a few special cases?
Even on sites that have a "Like / Don't like" button, my understanding is that clicking "Don't like" is a form of "engagement", that the suggestion algorithm are going to reward.
Give me a button that says "this article was a scam", and have the publisher give the advertisement money back. Of better yet, give the advertisement money to charity / public services / whatever.
Take a cut of the money being transfered, charge the publishers for being able to get a "clickbait free" green mark if they implement the scheme.
Track the kind of articles that generate the most clickbait-angry comment. Sell back the data.
There might a business model.
What could work is social media giving people an easy button to block links to specific websites from appearing in their feed, or something along those lines. It’s a nice user feature, and having every clickbait article be a chance someone will choose to never see your website again could actually reign in some of the nonsense.
Agreed, of course.
In a reasonable world, that could be considered part of the basic, law mandated requirements. It would be blurry and subject to interpretation to decide what is clickbait or not, just like libel or defamation - good thing we're only a few hundred years away from someone reinventing a device to handle that, called "independent judges".
In the meantime, I suppose you would have to bring some "unreasonable" thing to it, like "brands like to have green logos on their sites to brag" ?
> What could work is social media giving people an easy button to block links to specific websites from appearing in their feed, or something along those lines.
I completely agree. It's a feature they have had the technology to implement such a thing since forever, and they've decided against it since forever.
However I wonder if that's something a browser extension could handle ? A merge of AdBlock and "saved you a click" that displays the "boring" content of the link when you hoveron a clickbaity link ?
Reader: do the moral thing and read the article, not just the title
How is that balanced.
Should be able to go more efficient as the brain has other constraints such as working at 36.7 degrees C etc.
And, two, because the actual energy cost savings claimed aren't even the experimental question -- the energy cost differences between various operations on modern hardware have been established in other research, the experimental issue here was whether the mathematical technique that enables using the lower energy cost operations performs competitively on output quality with existing implementations when substituted in for LLM inference.
"The first release of bitnet.cpp is to support inference on CPUs. bitnet.cpp achieves speedups of 1.37x to 5.07x on ARM CPUs, with larger models experiencing greater performance gains. Additionally, it reduces energy consumption by 55.4% to 70.0%, further boosting overall efficiency. On x86 CPUs, speedups range from 2.37x to 6.17x with energy reductions between 71.9% to 82.2%. Furthermore, bitnet.cpp can run a 100B BitNet b1.58 model on a single CPU, achieving speeds comparable to human reading (5-7 tokens per second), significantly enhancing the potential for running LLMs on local devices. More details will be provided soon."
Nvidia will be very unhappy.
Obviously, energy cost creates a barrier to entry, so reduction of cost reduces the barrier to entry... which adds more players... which increases demand.
In other sensationalized words: "AI engineers can claim new algorithm allows them to fit GPT-5 in an RTX5090 running at 600 watts."
I had assumed the latency etc were based on what was desirable for the use case and hardware, rather than power consumption.
Maybe even generate a table of the approximate results and use that, in various stages? Like the way sin/cos was done 30y ago before FP coprocessors arrived
For he sake of the climate and environment it would be nice to be true.
Bad news for Nvidia. “Sell your stock” bad.
Does it come with a demonstration?
People say this but then the fastest and most-used implementation of these optimizations is always written in CUDA. If this turns out to not be a hoax, I wouldn't be surprised to see Nvidia prices jump in correlation.
If this increases integer demand and decreases floating point demand, that moderately changes future product design and doesn't do much else.
Surely even the OpenAI devs must have done this like the minute they got done training that model, right? I wonder if they'd even admit it was an AI that came up with the solution rather than just publishing it, and taking credit. haha.
https://hachyderm.io/@inthehands/112006855076082650
> You might be surprised to learn that I actually think LLMs have the potential to be not only fun but genuinely useful. “Show me some bullshit that would be typical in this context” can be a genuinely helpful question to have answered, in code and in natural language — for brainstorming, for seeing common conventions in an unfamiliar context, for having something crappy to react to.
> Alas, that does not remotely resemble how people are pitching this technology.