Hacker News Reader

Comment by ubedan | Hacker News Reader

Spectacular... Reminds me of the SimdJson algorithm in terms of jaw dropping nearly unbelievable speeds through creative programming. I hope this code get popular, as it will save tons of electricity, money, CO2, etc.

Have you considered publishing a rust crate as well? (If not, I volunteer.)

View on HN

> I hope this code get popular, as it will save tons of electricity, money, CO2, etc.

I don't think tokenization has ever been a meaningful bottleneck. JSON being fast falls into the same bucket much of the time. We spend way more energy on I/O and storage than we do on serialization and tokenization.

If you are concerned with economics and the environment, request batching would make a bigger impact. The most expensive part of this whole thing is GPU underutilization. You can save 50% with OAI right now if you can figure out how to make your workload fit the batch pattern. Do your users always need answers right now or can we afford to wait a few days in some cases? Tool calling doesn't "time out". Wall clock does not exist in the LLM. It took me a while to get used to this.

According to Jevons paradox this will likely lead to more electricity used and more CO2 being released to the atmosphere, because it makes it more profitable to build another data center.

Yes! I will publish a Rust crate soon. If you have thoughts about how to structure the API I would love to hear them.

Is there any write up regarding the SimdJson Algo? Definitely love to read more of it!

Github: https://github.com/simdjson/simdjson

It showcases an especially ingenious scheme for escaping json strings, as well as the other tokens necessary to parse json.

Daniel Lemire - One of the authors at QCon 2019: https://www.youtube.com/watch?v=wlvKAT7SZIQ

Another Youtube video that explains it: https://www.youtube.com/watch?v=vd9J9PPmAMM

Reply on news.ycombinator.com