Having a 20GB file that lets you ask an offline computer any question is amazing | Hacker News Reader

Having a 20GB file that lets you ask an offline computer any question is amazing

old.reddit.com

231 pointsbehnamoh122 comments

I am completely convinced the future is local LLMs, privet, on your device. But then also fine tuned with your own data. All your documents, email, movements, browser history - maybe it even gets trained on your calls. Effectively an extension to your own brain.

I really don't want my AI assistant in the cloud, controlled by a corporation, reading and seeing everything I do. I would go as far as saying I want local AI more and open source AI.

Another comment said they want it to be an offline Wikipedia, we don't need that, we need it to know all our own stuff, that's what we work with every day.

I also don’t want my AI in the cloud favoring a corporation’s values and goals.

But these tools are almost certainly going to come from them. And they want you to store all your data in the cloud too. And they want to keep you from running the software you want.

I think there is one company very well placed to do this, Apple. Their revenue is driven by hardware sales, rather than advertising, and they have committed to some level of "privacy" focussed design (although obviously there are criticisms). On top of that their hardware is looking almost like it was design for exactly these use cases. Apple silicone with its unified memory architecture, GPU and Neural cores is very well placed for local LLMs.

Apple have a history of doing stuff locally, on the hardware they designed, that other companies have offloaded to the cloud.

They are looking for "the next big thing" that lets them sell hardware. Some people thinks that's going to be their VR/AR headset, I don't. It's going to be local AI running privately on your device. Putting the AI in the cloud doesn't help with hardware sales.

My only hesitation is that they have such a crap spell checker, surly you would solve first...

> Apple silicone with its unified memory architecture, GPU and Neural cores is very well placed for local LLMs.

Yes, university research from 2011 already showed a privacy focused design to machine learning[1]. But will this be private or public like Wikipedia, Bitcoin and Bittorrent?

> Effectively an extension to your own brain.

Correct. After 1996 fantasy/visionary publications about this, we now know roughly how to do this. Most difficult problem is who should own it, no corporation in control is desired. That means full decentralisation, federated learning is not enough.

But fully decentralised learning is hard. Try to apply machine learning in permissionless, byzantine, unsupervised, decentralised, adversarial, continuous learning context. See my lab at Delft University focused for a decade already on "The Global Brain"[2]. With 2.3 million download and crowd-sourcing, we might get there..

[1] "Gossip Learning with Linear Models on Fully Distributed Data", 2011, https://arxiv.org/abs/1109.1396

[2] The Global Brain - the roadmap, https://github.com/Tribler/tribler/issues/7064

I think Apple will be hesitant to release an AI on their platforms that can be made to say unpredictably bad things. For now as far as I have seen there’s only censoring and that’ll only lead to a game of people trying to work around it.

> Apple have a history of doing stuff locally, on the hardware they designed, that other companies have offloaded to the cloud.

Looking at how they promote using iCloud on iDevices, I doubt that.

I don't think iCloud is a great example, because there's a definite value add in having your files synced between multiple devices and your computer. A better example might be Siri, which does basically nothing without an internet connection.

iCloud provides a lot of value, because storage is extremely expensive on all iDevices, and you can't upgrade it. Also Syncthing can be used for a synchronization between devices, if you control them.

Pretty every Google, Microsoft, Amazon, Samsung and apple can do that.

Everyone has a huge motivation to get this solved.

But because all of them also have their cloud storage and every customer is trusting their company I'm pretty sure most will not use local ai.

Until when suddenly in 10 years we have hardware chips like gpt-8 hardware version or something.

Eh. Patience. In the 70s, text editors, operating systems, and just about everything came from corporations, governments, and perhaps most commonly, universities when it came to software tools, and was hosted on their systems. As the hardware became more accessible, and the power needs diminished, it moved out from the mainframe to the client side.

This process will take time, but if we don't have the equivalent of chatGPT available on a home system (if not phone sized device) in 20 years, I'll eat my hat. There are already open source projects on Github that'll let you roll your own, but good luck coming up with the computing power.

And yet more recently everything began shifting to the cloud (storage, compute). Centralization leads to better economies of scale and therefore more efficiency.

It’s not clear to me this will ever change. While local compute will always improve (so local models become more viable), I believe that the largest and most capable models will always be cloud based.

If efficiency was the only metric, then I agree. However, we don't all need the largest and most capable models all the time. For example, stockfish running on my phone would destroy me and almost every human on Earth, even if every human collaborated together to try to beat it. Also, centralization isn't always the panacea people make it out to be. Google search was very efficient and centralized. They're about to have their lunch eaten by a start-up from nowhere. McDonald's hamburger is very efficient, but I eat that sparingly. Varieties and different (inefficient) paths are still viable and usually needed to progress further. In many cases, centralization in the name of efficiency usually spells the beginning of the end for many firms (and countries).

https://mitta.us is one project that I find myself attracted to but haven't dare to try because it isn't local.

It does 25% of what you ask (subtracted 50% because it isn't local and 25% because it is missing calls, movies etc).

I wonder if at some point there will be an open source project like this?

Computers are becoming smaller and the internet is becoming even more ubiquous. I don't see your future happening.

Why would I spend a lot of money, space, and maintenance time for a computer with a good GPU when I can just use any of the two internet connections I'm already paying for? I'm already trusting corporations to read my email, and no ChatGPT answer can get more personal than that.

This is also why I think cloud gaming is only going to become more popular.

Cloud gaming was a crash and burn story already.

There's a stronger resistance to cloud than ever before after all the leaks, hacks and pushed agendas.

These models aren't cheap to host and they'll lose to free local models that don't have boundaries or agendas. Cloud AIs inevitably will try and monetise by selling you crap or be funded to shape your opinions.

> Cloud gaming was a crash and burn story already.

What? No it isn’t. One company failing doesn’t mean the industry has crashed and burned. I use Xbox cloud gaming all the time along with millions of other people.

Even Microsoft has admitted (between the lines) that cloud gaming isn't as profitable as just selling games. And it's not just one company that's failed either. Google, Onlive, Gaikai, there was a company that made a Cloud console a couple years back, the list goes on. Besides, the latency will never be low enough, and the bandwidth cheap enough, for a lot of people to make it a viable solution.

Gaming is a total different field.

Hard to make any assumptions out of this.most people I know definitely sync there data onto companies cloud storage.

Alone for image backups and sharing and etc. No one minds that.

No one minds mail analysis either.

Cloud is healthy, very healthy

It is, but AI is a different kettle of fish altogether.

We already push back on targeted advertising and tracking, think what AI is capable of with your data. If its writing your emails, if its summarising for you, if its consulting for you...

It can push agendas and influence. Its quite dangerous.

These models need to be local as much as possible and they need to be better than what the cloud can offer.

At least for me this isnt one the decentralised Internet can take if we don't want to go down the dystopian future route.

> There's a stronger resistance to cloud than ever before after all the leaks, hacks and pushed agendas.

It's a thing on HN loop.

I remember google desktop search. It was amazing, even today's win 11 search doesnt come close. So something snappy like that with LLM will be perfect.

Yes! I absolutely loved Google desktop search and hoarded much more pdfs and tex files than otherwise. Very excited for the LLM equivalent.

PrivateGPT, although more of a proof of concept right now, looks like it could be a nice fit in future

Yes, I whole-heartedly agree. What we need is a proliferation of AI tools. Centralization of compute in the hands of a few corporations could be worse than monoculture farming. If we don't create, mix, and match our own models we're offloading all decisions to opaque companies like the "open" OpenAI.

So a hackers main goal will be to get your LLM data, and companies will be backdooring them?

Yeah, I would definitely be leaning into the "offline" word in the original title for something like this.

Why would this be a risk? The data is already on your machine. It would be much easier for a hacker to target that data directly, as they do today, than to try to extract some impossibly obfuscated form of it from an LLM.

Well if they are interested in data. If they are interested in asking a prompt "hey what's my cc number and password again?" That'd bypass an awful lot of encryption. Would also be easy to find blackmail for a lot of people thinking it's a secure way to lookup crimes.

These things will inevitably be included with the OS soon

Desktop, mobile, even consoles - especially video game consoles where games can tap into the shared resource for their NPCs instead of bundling it in their package

GPU based token generation will be good enough in speed

It's probably the future you want. But fact is, most of peoples emails, documents, browser history, travel history, bookmarks, contacts, etc., already live in the cloud. People are obviously not concerned with it, so I would say the future of LLM's is in the cloud using all of the cloud data.

I was pretty impressed by the power and accuracy of the Google Translate offline models (which predate some of the amazing new LLMs), also in a seemingly amazingly small space—tens of megabytes per language, if I remember correctly?

Firefox's builtin translation is not bad either. I've seen it make a mistake here and there, or fail to translate certain words and expressions but it works generally well.

Is it an offline system, not using an API?

Yes.

"The Firefox Translations extension can automatically translate content from the web pages you visit. Unlike some cloud-based alternatives, this extension translates text locally in Firefox, so the content you’re translating doesn’t leave your machine."

https://addons.mozilla.org/en-GB/firefox/addon/firefox-trans...

Cool, I'll give that a try! I didn't know about it before.

I feel the same way with running stable diffusion on my MacBook (DiffuseBee app). The idea that I can generate an image of almost anything and it has references to what it means in the real world, and carry that in my backpack, is little hard to comprehend.

My goal: download the entire Wikipedia and store it in a vector db. Then have a local LLM answer questions based on Wikipedia. Best of both worlds!

I wonder what personality it would take on, when you ask it write fiction

Given the biased nature of many if not most Wikipedia articles it would be certain that such a system would bend the truth to an unacceptable degree. Train the thing on more than just Wikipedia, e.g add Encyclopedia Britannica to the training set. This is the one area where "diversity" really matters: diversity of opinion. Using a single biased source as your oracle will turn you into a pawn for those who control the bias.

what? since when wikipedia is biased? and about what? (and why EB isn't?)

At the risk of conflict of interest, there's a wikipedia page that acknowledges that niche sections of the site have biases. Of particular interest is the section on Croatia.

https://en.m.wikipedia.org/wiki/Ideological_bias_on_Wikipedi...

It baffles me how you can either have a news/information source with a left-wing bias or a total cesspool of untrustworthy information with a glaringly obvious agenda.

This comment seems to get some hate, but all you have to do is skim over the linked Wikipedia article to verify what I'm saying.

I agree, and for another demonstration, I invite everyone to look at similar sites as Wikipedia. They are either "left wing bias" (RationalWiki) or one with an agenda, divorced from reality (Conservapedia). It almost looks like that if you organize basic democratic cooperation among people, that's automatically "left wing" from where our current narrative stands.

Some parts of RationalWiki aren't that rational any more, unfortunately.

Eh, if most is rational, then I think they do a good job. Very factual too, which is good to know:

https://mediabiasfactcheck.com/rationalwiki/

> They are either "left wing bias" (RationalWiki) or one with an agenda, divorced from reality (Conservapedia).

   #/bin/env bash
   read -p "Please enter your personal bias (e.g. conservative, liberal, progressive, etc.) philisophy: " bias
   echo "they are either ${bias}ly biased or with an agenda, divorced from reality."

Do you think the same about OP's statement that I reflected on, stating "you can either have a news/information source with a left-wing bias or a total cesspool of untrustworthy information with a glaringly obvious agenda"?

As a point of reference, if you search high traffic websites with "Left bias" and also with "Right bias"[0], it will turn out that the average factuality of the sources are higher on the "Left bias" news sources.

This makes me think that the point of reference, the "center", is now off-center, to the right. To the point where actual reality seems "left". "Facts Have a Well-Known Liberal Bias". Considering this, sources even further to the right will of course have worse information, same as they would on the far left. And this is for news sources - what I added was that I observed the same for wikis.

[0] https://mediabiasfactcheck.com/filtered-search/?bias=Left&tr...

That's a cute way to say essentially nothing.

Edit: feel free to defend the Conservapedia's take on climate change for example https://www.conservapedia.com/Climate_change. Absolutely laughable bullshit.

While it was not said out loud, it reminded me of the "both sides" fallacy. Like how schools should teach both sides, creationism and evolution, as if both have the same merit. Likewise, the bash script implies that bias is bias.

https://en.wikipedia.org/wiki/False_balance

US politics, definitely. But also any other topic that attracts an internet mob large enough to infiltrate the editor community.

Take the first few paragraphs of Donald Trump [0] vs. Joe Biden [1] and try to compare the way they're presented:

    Donald John Trump [...] and his businesses have been involved in more than 4,000 state and federal legal actions, including six business bankruptcies. Trump promoted conspiracy theories and made many false and misleading statements during his campaigns and presidency, to a degree unprecedented in American politics. Many of his comments and actions have been characterized as racially charged or racist and many as misogynistic. [...] He reacted slowly to the COVID-19 pandemic, ignored or contradicted many recommendations from health officials, used political pressure to interfere with testing efforts, and spread misinformation about unproven treatments.

There's not a single positive sentence about him. He must have done something positive in his 4 years of reign, but Wikipedia doesn't tell me that in an objective manner. Meanwhile...

    Joseph Robinette Biden Jr. [...] has addressed the COVID-19 pandemic and subsequent recession. He signed the American Rescue Plan Act, the bipartisan Infrastructure Investment and Jobs Act, the Inflation Reduction Act, the CHIPS and Science Act, and the Respect for Marriage Act, which codified protections for same-sex marriage and repealed the Defense of Marriage Act. He appointed Ketanji Brown Jackson to the Supreme Court. In foreign policy, Biden restored America's membership in the Paris Agreement on climate change.

There's not a single negative sentence about him. He must have done something negative in his 4 years of reign, but Wikipedia doesn't tell me that in an objective manner.

Whenever someone tries to portray someone as completely evil or completely saint-like, I call bullshit.

[0] https://en.wikipedia.org/wiki/Donald_Trump

[1] https://en.wikipedia.org/wiki/Joe_Biden

The article covers allegations of corruption and mentions his "gaffes" a number of times.

Sometimes where there's no smoke, there's no fire. The guy is very boring and has a weird kid. That's about it.

i know you're being sarcastic, but do i think its interesting that centrist politics is sometimes viewed as less corrupt just because the bias is harder to articulate

Regarding centrists:

* https://en.wikipedia.org/wiki/Radical_centrism

One POV on it:

* https://www.theguardian.com/commentisfree/2021/may/28/centri...

Exactly. This is the tip of the iceberg.

Your [...] are very convenient, because you skipped over the part in paragraph 3 of Trump's article where they go over some of his main "achievements", which are presented neutrally and some people might consider positive:

    [Trump] signed the Tax Cuts and Jobs Act of 2017, which cut taxes for individuals and businesses and rescinded the individual health insurance mandate penalty of the Affordable Care Act. He appointed 54 federal appellate judges and three U.S. Supreme Court justices. He initiated a trade war with China and withdrew the U.S. from the proposed Trans-Pacific Partnership trade agreement, the Paris Agreement on climate change, and the Iran nuclear deal.

and you also cut out a mention of a pretty significant Biden failure (in outcome, at least):

    [Biden] oversaw the withdrawal of U.S. troops from Afghanistan that had been negotiated and begun under the previous administration, ending the war in Afghanistan, during which the Afghan government collapsed and the Taliban seized control.

But moreover, the introductory paragraphs of a Wikipedia article, like the introduction to most pieces of writing, are meant to establish an overview. Assuming that the reader reads nothing else, what basic idea should they have of the thing?

Donald Trump spent his four years in office much like he spent most of his previous business career: lying and committing various forms of fraud. While president, just for variety, he added lying and committing fraud about COVID, and, oh yes, inciting a violent insurrection to overthrow the American government. That should be the first thing you take away. He may have accidentally done some positive things, but none of them make the lead paragraph, because they won't be in the top 3-5 things that history remembers him for. "Scholars and historians rank Trump as one of the worst presidents in American history" appears near the end of the introduction (right before jumping back into more recent events), because that's a germane summary.

Joe Biden has spent his 2+ years in office being a relatively uninteresting mainstream-to-slightly-progressive Democrat. That should be the first thing you take away. He's not perfect, but he's doing normal president stuff. Once he's no longer president, no doubt his article will be updated to look like Obama's, which currently ends the introduction with:

    Rankings by scholars and historians, in which [Obama] has been featured since 2010, place him in the middle to upper tier of American presidents.

...that is, not saintlike, but, you know, fine.

> Donald Trump spent his four years in office much like he spent most of his previous business career: lying and committing various forms of fraud. While president, just for variety, he added lying and committing fraud about COVID, and, oh yes, inciting a violent insurrection to overthrow the American government.

Interesting that I can't find any reference for such claims in the Wikipedia article. The whole introductory paragraph contains the total of 2 sources - one a politico.com article [0] and the other a Siena College Research Institute post [1] - neither of which provide any claims other than that Trump was unpopular and "rated as worst".

You can't with a straight face claim that this is unbiased writing.

[0] https://www.politico.com/news/2021/06/30/trump-cspan-preside...

[1] https://scri.siena.edu/2022/06/22/american-presidents-greate...

What do you believe happened on January 6, 2021, in Washington DC?

I mean trump is a gigantic clusterfuck.

Why is this not objective enough?

This is actually a pretty sound summary of the two. The one guy just got charged for rape, recently another one of his companies went down for fraud, he invited a riot on j6 to overthrow democracy, is under investigation for stealing hundreds of classified documents, lied tens of thousands of times(people take the time to count these things with references for presidents), issued dozens of legal challenges to try to overthrow democracy all of which were found to be based on nothing across the country, called racists dressed up as Nazis fine people, defended adversarial dictators, ate classified materials, national archives had to pull some out of toilets, etc.

The other guy mostly eats ice cream and hangs out in delaware... They tried investigating him already a few times and his son and came up empty handed. Maybe time will prove he's not the best guy ever, but it's pretty safe to say the other guy should be in jail because he's a prolific criminal.

Award for best understated comedy on this thread.

Really? Wikipedia's left-wing bias is well known:

https://www.foxnews.com/politics/wikipedia-bias-socialism-pa...

https://twitter.com/elonmusk/status/1600200000763756544

https://www.dailymail.co.uk/news/article-9283061/Wikipedia-f...

https://nypost.com/2021/07/16/wikipedia-co-founder-says-site...

https://www.dailysignal.com/2022/04/27/wikipedias-left-wing-...

Compared to Britannica:

https://hbswk.hbs.edu/item/is-wikipedia-more-biased-than-enc...

Example: https://nitter.net/echetus/status/1579776106034757633#m

Wikipedia is as biased as the writers of the articles. Wikipedia has articles outlining bias in its own articles. These things are authored by people, and people are biased. Wikipedia knows this and so should its readers. They aren't unique.

> since when wikipedia is biased

Since the first article hit the site which clearly showed off the personal bias of the author. Then since the first article was edited by a group of like-minded editors who approved of each others edits but were vigilant in keeping other opinions off the page. These groups of editors come in different flavours ranging from ideologically driven [1] - these control the narrative on pages related to politics and politically sensitive topics like SARS2 and "Climate" - to commercially employed to keep unwanted narratives off certain pages.

> and why EB isn't?

Who says that Encyclopedia Britannica is not biased? They are to a certain (although lesser) extent, they just happen to be differently biased. That is where my remark on the value of diversity of opinion comes in.

[1] https://thecritic.co.uk/the-left-wing-bias-of-wikipedia/

I would probably also try to add more sources than Wikipedia but I don't think your critic has any day to day consequences at all.

yes someone please annotate/convert wikipedia and make it available as torrent!!

https://meta.wikimedia.org/wiki/Data_dump_torrents

* in vector encoding

There's this by Cohere - https://txt.cohere.com/embedding-archives-wikipedia/

One of the comments on reddit

> i have no clue why everyone isn't talking about this all the time.

Talking maybe, not doing it because it's still too out of reach for most of us? I've got a laptop with 32 GB RAM and a 2GB NVIDIA card. At which speed can I run that model, if I can run it at all?

2GB VRAM means you can run things comparable to GPT2, a glorified Markov chain. On your CPU you could run much larger models at far from real time speeds.

2gb vram is not enough for running LLM but you can run on CPU gpt4all provides gui just click click download model click and you are ready, just give it a try

start with `mpt-chat7b`, my suggestion

https://gpt4all.io/index.html

Thank you for that, I’m installing now

I currently have a copy of offline Wikipedia (and Stack Overflow, WikiVoyage, etc.) via Kiwix on my laptop and also LLaMA-style models at the same time (since I like things being local), and with respect to asking an LLM questions, it would be great if it could leverage the offline Wikipedias to ensure it stays grounded and not make up facts.

This is why I also like to ask phind.com and chat-4. Helps keep things grounded.

How much space does that take up, especially SO? There's so much junk on there.

Around 159 GB, it's surprisingly small.

We can do that, but then we will need to close all applications to run it because it will need 20GB RAM or VRAM to run.

It is still more practical to visit stackoverflow when I have a question.

(still impressive how much this improves every week. Maybe in 2-3 months we have something more useful)

I got 64gb ram in my desktop for few hundred €. Laptops overprice RAM (as a price discriminator I guess).

But unlike Apple SoC, most GPUs have separate memory - getting a GPU with 24gb ram is very very expensive in comparison.

I just upgraded my 2020 era thinkpad (t470, which I bought this time last year for less than £300) to 32g memory for £70.

20g of ram doesn’t feel like a lot today.

Also today: soldered memory and 8GB MBPs that START at $1300.

The magic of swap means you can leave those applications open and as long as they're idle they'll just quietly wait for you on disk until you've finished your LLM adventure.

On many modern systems swap is quite small and not large enough to hold the entire system. For a 32GB machine, Ubuntu recommends 6GB and redhat recommends 6.4GB (20%). In a VM that number is much much lower. Besides, some libraries allocate memory in such way that you get artificial memory spikes in physical memory.

I know this because I spent this weekend debugging crashes in an LLM pipeline :(

Ubuntu recommends a /minimum/ of 6GB, not sure it's fair to say they recommend you actually use the minimum though.

Why would I do something like that?? Every time I ask something to chatGPT I get wrong answers…

Maybe you’re asking the wrong questions

He knows the answer but not the question

Correct. Parent commenter seems to be asking questions they already know the answer to.

Those are the wrong kind of questions.

They should only ask the questions about things they don't know anything about.

ChatGPT gets those right every time.

I appreciate the facetiousness.

This is the funniest yet most meaningful comment I've read this year!

fun laws - the Gell-Mann amnesia effect / Knoll's law of media accuracy:

https://en.wikipedia.org/wiki/Michael_Crichton#GellMannAmnes...

I don't know about every time but honestly the fact that it'll just fill in the gaps when it doesn't know means that you have to treat all responses with a bit of suspicion.

"Computers are useless. They can only give you answers." — Pablo Picasso

* https://quoteinvestigator.com/2011/11/05/computers-useless/

You can download a local version of Wikipedia and yet barely anyone does this. (And I know wikipedia has a ton of problems)

Or an Encyclopedia Britannica on the shelf/in pdf. People don't know that they need it, but it indeed feels amazing when you need an information, and it is there.

Wikipedia is static, a local AI can be customized and adapted to your needs.

For example I would love an AI that could filter, summarize and prioritize news that I'm interested in.

AI doesn't know anything past September 2021. Now who is static?

ChatGPT doesn't know anything past September 2021.

Local LLMs can be as recent as yesterday.

Plus they can access a daily updated offline version of Wikipedia as well as news RSS feeds via LLamaIndex and/or Langchain.

Yeah it's a pretty fun dataset that you can do a lot of stuff with. Very slept on.

Now I have an excuse to purchase high powered machines. As a programmer who doesn't play games, edit video, or do anything requiring a high-powered graphics card, I always eyed high-end rigs as a wasteful indulgence. Not anymore! Now I can't wait to train my own models on my own data and make the computer do things for me, like its 1982 all over again.

I recently purchased my first new computer since 2008 — a Mac Mini M2Pro, stock.

It's incredible what this $1300 machine can do, almost out of the box (e.g. DiffusionBee).

From my understanding the device shares RAM/VRAM, and has a total of 16GB (up to 32), 14.5~GB of which can be used for LLM/genAI processess.

I am just a layperson, a non-gamer; and this purchase has been really wild for local generations.

The thing about transformer models that are impressive is that they compress all sorts of stuff about human understanding down into a incredibly tiny space. They asked GPT-4 to explain GPT-2 and it found things like "something done right" or "metaphors" corresponded to neurons. It's this meta understanding of thought, just like diffusion models are this meta understanding of art. The real innovation is one can turn a crank and it keeps getting deeper and more subtle insights into the whole thing. If you turn that crank with A100s for a few days it gets very deep and doesn't hit a wall. In the earlier days of machine learning that didn't used to happen. Understanding was bounded at some point and didn't increase, but it seems transformers just keep going.

I think this will be more mainstream if someone create an installable app (win/mac/linux), right now I did a bit of research and I have to install several stuff (with https://huggingface.co/), it was confusing to a non advance user, I wasn't able to do it.

The attitude in the comments is amazing. The LLM chatbot hype cycle bears a lot of resemblance to the crypto hype before the crash.

The utility of LLM's is a very different question as the profitablity of LLM's.

That was something that was severely overlapping with crypto. Most people wanted to believe it's useful because it would mean $$$

LLM 's are here to stay, that's for sure.

It seems like everyone in our company uses generative AI at the moment: marketing department coming up with ideas, designers creating illustrations, programmers writing code etc. I.e. there's a lot of useful real life application of generative AI right now, people actually use it for work. Crypto, for most part, in my opinion, did not provide as much value to the average person. And it's now possible to run LLMs locally, so you shouldn't be afraid of exit scams :)

I would pay several hundred dollars a month to keep access to GPT-4, and never found any use at all for crypto. There might be hype, but there’s also an insane amount of utility

Some examples please. I’ve yet to find any actual utility, mostly I’ve got a lot of convincing bullshit out of it.

I use it to speed up writing boilerplate code. That alone saves me enough time to justify paying for access many times over.

For almost anything, I find it a better Google, and I can go fact-check it afterwards.

It’s a better StackOverflow and better than most project or dev docs for me: I’m an experienced developer who’s writing Python for the first time. I know what I want to achieve each time (how do I produce the Typing type for…) but I find it much easier to use GPT.

I use it to search for books or films or video games where I half remember the details, and it gives me options.

I recently finished a dissertation (on GPT, as it turns out) and it was useful for me to research sensible correlation types I could use and build out my understanding (“wait, is Spearman’s rank is just Pearson in the rank order?”). Also I could copy and paste sentences I didn’t understand about methodologies in papers I was reading, or ask it abstract questions like “is there a standard name for a modified Likert scale but with a ‘not enough data’ option”.

I was really struggling to work out how to change just one page orientation in Word, on a Mac. First few Google results I had talked about the Windows version, which had features mine didn’t have. GPT put me onto section breaks.

Was Caffè Nero named after the emperor, checking my intuition about how short-term SSH auth certs work, jq recipes, why were Man U so popular when I was a kid, why did they suddenly start winning more often?

I am curious about a lot of things, and GPT provides me instant answers without having to click through bullshit. If it’s something it needs to be right about I can double-check it’s answers. I am averaging ten topics a day on it, most have a bunch of follow-up questions.

For me it's not the answer but the formatting or presentation of it.

So it's like a normal search but where you don't need to reformat and rearrange the result to suit your needs.

Within a few minutes you can sign up to OpenAI and see the potential of ChatGPT immediately, in you own hands. The potential of crypto/web3 was always an abstract idea (jumped on by so many scammers), there was nothing you could actually use to see the utility.

Plus there is tooling like Copilot that works and is useful. I would love to have something like copilot integrated across my whole system, gathering all my current context, like terminal, notes, mail, issue tracker and have even just plain, simple autocomplete everywhere. It is feasible, and I'm pretty sure we will have more and more integration. I was sceptical, but copilot changed my mind.

Though there are people selling pickaxes at gold rush, it's not pump and dump scheme like most crypto

LLM has provided real value for me the last couple of months, both as amusement, as a way to "talk" while working from home and by solving problems.

And I was late to the party and arrived skeptical.

With crypto I arrived early (2009) and in hindsight realized that the only real value I can see from most of its existence is transfer of money to and from hard-to-reach-financially places and as an object of financial speculation.

Learning is going to be revolutionized.

Yes, this. Once they get a smidge better and can answer questions? Every student will have the King's Tutor - it will be Blooms Two Sigma in action.

...which will drive down the cost of all knowledge work and rather than reconfigure our society we'll simply have no entry level jobs anymore and the few people who still have jobs will still be expected to work full time but have insane outputs.

This whole thing is a Monkey's Paw.

Why have a job when you have all the knowledge to sustainably live off the grid?

I expect self-employment to skyrocket.

You had all that knowledge before too, free from the internet. It’s just easier and common thing to do to live within the system with all the other people. But this is changing also with the changing world conditions.

How restricted are these models? ChatGPT often refuses even simple questions

What's the best model for 16GB GPU on my laptop?

WizardLM-13B-Uncensored

thank you!

Reply on news.ycombinator.com