Hacker News Reader

Comment by dmix | Hacker News Reader

I wonder how many LoC that is on average. Is there an average for LoC? It’s probably based on the language…

Testing some random rust code, it's about 15-20 tokens per LoC, so about 1500-2000 LoC in a 32k context.

Interestingly, using 2-space indentation, as soon as you are about 3-4 indentation levels deep you spend as many tokens on indentation as on the actual code. For example, "log::LevelFilter::Info" is 6 tokens, same as 6 consecutive spaces. There are probably a lot of easy gains here reformatting your code to use longer lines or maybe no indentation at all.

Make sure you are testing with the tiktokenizer: https://news.ycombinator.com/item?id=35460648

Ah, good catch, it's actually closer to 8 tokens per LoC with GPT4's toktoken, so about twice as good. Some quick testing suggests that's mostly down to better whitespace handling.

> There are probably a lot of easy gains here reformatting your code to use longer lines or maybe no indentation at all.

Using tabs for indentation needs fewer tokens than using multiple spaces for indentation.

Curious how this point in the tabs vs. spaces debate re-emerges

Finally, APL's day is here! https://en.m.wikipedia.org/wiki/APL_(programming_language)

How do this even work if the code also uses external libraries?

A LoC is ~7 tokens thanks to the new toktoken tokenization in GPT-4.

32k is ~4.5k LoC

Reply on news.ycombinator.com