Hacker News Reader

Comment by praveen9920 | Hacker News Reader

Imagine an LLM that is trained on public data like the internet archive and it is made available on these boxes without the internet, now we can have almost the entirety of human knowledge accessible on a single box.

I know that LLMs sometime hallucinate and practical limits of computation power availability etc. but imagine how quickly we can spread the knowledge. It is almost like the printing press during the renaissance.

View on HN

LLMs are not magic.

If wikipedia takes up 100GB of space and your model trained on it is only 5GB big, there is NO WAY it will hold all the original information.

NNCP, a state-of-the-art lossless text compressor based on a transformer neural network (i.e. containing similar structures to a LLM), can reach a compression ratio of 10.8% (0.867 bits per byte) on Wikipedia text: https://bellard.org/nncp/

Compared to NNCP, a hypothetical LLM that "knows" every fact in the English Wikipedia, would not need to keep a lossless record of the entire text of Wikipedia, since many Wikipedia pages repeat facts that also occur on other pages, and instead of reproducing every word it would be enough to paraphrase the ideas. Thus the size of such a LLM could be much smaller than 10% of the size of Wikipedia. How much, it's hard to say...

Yeah neural nets are amazing compression algorithms since the best way to get the gist of something is to actually understand it, then algorithmically generating the data. Like a calculator having a near infinite compression ratio because it can do math instead of keeping a table of all possible inputs and outputs.

Same goes for human brains I suppose, we can only remember as much as we do because of the effective compression our understanding allows for.

Furthermore, LLMs are good at condensing information, but terrible at knowing which information will actually be important or true. This is an obvious problem in a life or death medical situations.

…which is also unsolved by any other system or by humans. It’s a fundamental constraint of reality.

As of 21 September 2022, the size of the current version of all articles compressed is about 21.23 GB without media.

https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia

By the way, note that is the Wikipedia in English which AFAIK is the largest one. For instance, Wikipedia in Portuguese weighs about 13gb with a few pictures in my Kiwix install.

Has Kiwix gotten any better? I tried to use it a few years ago, but the user experience was super rough. Fantastic idea, but there were lots of UI stalls and glitches that soured my trial.

Honestly? The UX is dogshit lol, at least on the Android app, when you go and download something to find out if it has downloaded or not is a chore and there is no progress notification or indication it's doing anything at all.

I'm considering sending some contributions whenever I have time again for OSS contributions.

However, I find it usable enough that I've been using it quite a lot recently when I don't have internet access to read Wikipedia, StackOverflow and a few other things I downloaded.

I'm not sure this is true.

Lossless compression algorithms exist, and HTML compresses almost absurdly well. A 20x compression rate would be quite impressive, but I don't think it's impossible, since 10x compression is fairly common in this domain, and there are reasons to expect wikipedia to compress even better given the significant amount of markup on each page.

I've worked with cleaning up rendered wikipedia texts, and if you strip the mediawiki class names and cruft, the uncompressed HTML shrinks a lot, often from hundreds of kilobytes to tens of kilobytes. You can then probably compress that quite a lot as well since plain text compresses very well on its own.

Basically, mediawiki uses this much markup to say <li><a href="#Assessment">Assessment</a></li>:

  <li id="toc-Assessment"
  class="vector-toc-list-item vector-toc-level-1 vector-toc-list-item-expanded">
  <a class="vector-toc-link" href="#Assessment">
   <div class="vector-toc-text">
   <span class="vector-toc-numb">6</span>Assessment</div>
  </a>

  <ul id="toc-Assessment-sublist" class="vector-toc-list">
  </ul>
 </li>

So yeah. 20x isn't too far fetched.

That is not the point.

There is a physical limit of how much you can compress stuff. If your LLM is significantly smaller than even the most optimized compression we can achieve, then it simply doesn't actually preserve all the information

Indeed, if the theoretical limit to compress wikipedia is n GB, then you will need n GB at least to create the exact same copy of wikipedia. However you may create a jpg of it. (And you can then remove the artifacts, and even enhance it with AI, and you will get something better than the original wikipedia, to keep the image analogy.)

Be that as it may, 5Gb just isn't realistically significantly smaller than you could squeeze the text of wikipedia.

It doesn't need to be lossless to be useful. A lossy LLM can just throw all the redundant stuff away and focus on the actual information in the text, instead of the text word by word. It wouldn't need to keep dozens of languages around for if it could translate on the fly. And it wouldn't need to keep a full text when it can just have the raw facts and generate the text on the fly.

Not sure what the state of the art is with LLM text compression, but with StableDiffusion compression ratios of up to 155:1 have been achieved[1] compared to 10:1 for JPEG.

[1] https://pub.towardsai.net/stable-diffusion-based-image-compr...

Let's be honest disk space is cheap enough for people use actual data instead of LLM-generated.

What people in poor countries need the least is learn from source that have tons of random facts of knowledge censored out since some western corporation decided they're unetical or disturbing.

In my experience LLM's hallucinate a lot.

Often giving "plausible but entirely wrong" output that is hard to discern from factual, real output.

Yes.

At the moment they are great for entertainment.

But for factual information, you are better off just grabbing a copy of Wikipedia.

Wikipedia's information often isn't collated how you want. Eg List the states where the biggest city is larger in population than the state capital? There isn't specifically an article for that. Sure you could take the List of United States cities by population article and assemble the list yourself, or you could just have ChatGPT do it for you.

Yes, language models are already great for collating information, but they are still too unreliable to rely on.

Reply on news.ycombinator.com