I'm convinced now that “personal LLMs” are going to be a huge thing | Hacker News Reader

I'm convinced now that “personal LLMs” are going to be a huge thing

old.reddit.com

52 pointsbehnamoh51 comments

One reddit comment in opposition to this idea I agreed with was

"If Google turned on an LLM tomorrow that had access to your whole text message history, location history, calendar, google docs & drive, and all of your gmail (sent and received), almost everyone would start using it and keep using it."

https://old.reddit.com/r/LocalLLaMA/comments/16au3ga/im_conv...

So then all we need is Charlize Theron's voice in an all voice command world, and were there where Her brought us. The future came faster than I thought

that's multiple orders of magnitude more training data, I don't think it makes sense financialy (or possible in the near future).

I think the public opinion of LLMs is so severely damaged right now that if Google made that available tomorrow it would be a media circus of bad incidents, experts, and even politicians chiming in.

That the public has a poor understanding of technology cuts both ways. There's no such thing as user error if they're talking to an LLM. High expectations require high amounts of testing, regulation, etc.

That's both very true and very sad.

Users will get one wrong answer at a critical time and then google will shut it down.

Actually, they'll probably just shut it down after a couple of years.

If that was true, Google Assistant would have been archived years ago.

You mean if Google Assistant was backed by an LLM instead of whatever combination of tools is backing it now?

Yes, 100% this is correct. And its been fairly overtly the direction Google’s AI efforts have been directed at since long before “LLM” was the an implementation technology they could use for it. If Google was any good at productizing its AI efforts, it would already have happened.

Here's a fundamental truth that HNers can't or won't accept: Consumers don't care about privacy, at all. Maybe conceptually, they might object to Big Tech knowing as much as they do, but it doesn't affect their decisions in any way. They simply don't care that Google Maps is always tracking their location, that an Echo Show has a camera and mic that can be remotely activated, that TikTok is logging their viewing habits down to the millisecond, nor that their Smart TVs are logging their media choices. They just don't care. It doesn't affect their willingness to use or buy technology in the slightest. The comment you quoted and your agreement shows this quite plainly.

This actually bodes well for AI services, as they will be more useful the more they know about you. Your age, gender, sexual preferences, religion, work, family, possessions, money, tax records, account logins, schedule, habits, health records, etc. will help AIs to help you. This is the inevitable end game for AI services, so the fact that your average joe has no real conviction about privacy will hasten that future.

Side note: If your business or product plan includes "privacy" as one of its core features, expect limited success as you're targeting a very niche part of the market. No one cares.

Humans can tolerate increasing amounts of discomfort as long as it's incremented slowly or alternatives are blocked. Privacy loss is simply one of the many discomforts which we tolerate. Which is why pointing a flaw in any long used product of service causes such antagonistic reaction - that flaw was already normalized by us.

I don't know why I can't seem to get on the LLM hype train everyone is so ready to jump on. I either lack vision or see it for what it is, a next-gen search engine. And don't get me wrong that's amazing, but not world-shattering as people want to believe.

It's world-shattering for people who have never had a good way to manage their personal knowledge base. Personally, I just let my mind organize my thoughts. If something's important I'll remember it, if it's not then I don't. Obviously I write stuff down where needed but that's more to help commit it to memory, not because I'm ever going to read these notes again.

This resonates with me because personally I don't see a lot of use for my (dev) workflows. Was talking with my SO recently on this topic, they are close to finishing a PhD and do scientific research. For their workflows, consuming a lot of written information and producing a lot of written information, a case could be made it is more useful.

yeah, as a developer I've yet to see any development tasks that an LLM would be useful for, and it's wasting resources that could be going into abstract reasoning about software, or really doing anything correctly/accurately...

as a developer, I can see a lot of tasks that an LLM would be useful for. In a large organization, new people get onboarded all the time, and generally they have the same exact questions, like 'how do i do a web request out of the proxy?' and 'how can I do multithreading' or 'how do i use library x for our use case y'? whereas an LLM can look at the code used throughout the business and offer suggestions like "hey, I see you're trying to make an https connection to an external site without using a proxy, try this instead". Getting enterprise code up to the enterprise standard, whatever that is, can be very useful. Personally, I'd love to go in depth with an AI on crazy ideas I have for code, and work through a lot of ideas before I implement something. For example, I might want to try different ways to accomplish the same task. I could write my version of it, and instead of rewriting it to try a different way, I might ask an LLM to rewrite it that other way, so I can check which way is better.

So basically what the OP said, a next-gen search engine - trained on your code (rarely) or on SO (usually).

Those all sound like great things to ask of an assistant tool - that ChatGPT isn't actually capable of answering. (I have lots of those too - things like "review this code" or even better and more specific, "given the description in this CVE, do we have any code that does this sort of thing too, that we should examine"? Fortunately for the perceived level of honesty in the field, noone appears to be claiming an LLM can do either of them.)

Conversely, I routinely use LLMs to do development tasks now. My last bit of greenfield work, I had an LLM stub out an entire API for it. When I needed to marshal a fairly complex JSON object into a well-defined object in my code, including several validations, due to some security controls, the LLM taught me about a library I wasn't familiar with in this language.

Or, a few days ago, I had the need to run the same CLI commands a few dozen times with slightly differing parameters. Unfortunately, the CLI only exist in Windows, and needed to be called on multiple hosts in a Windows environment. I could probably do this on Linux using bash and/or Python, but in Windows, it was way easier to just have the LLM write a PowerShell script for me.

LLMs aren't the best at dealing with proprietary codebases (yet), but I'm sure that will come. In the meantime, they're really useful for abstracting away mundane work in a way that's much more user-friendly than your IDE probably offers, and they often help me spot issues with my assumptions, as well.

But what if you enjoy doing those things by hand?

Nothing wrong with that. I don't, though -- so LLMs have made me more productive with less annoyance.

(My personal struggle is figuring out when to stop trying to use the hammer that is LLMs. I've definitely fallen victim to the sunk cost fallacy here.)

hmm, is the powershell example something you don't expect to have to do again, so it's not worth really understanding the details? (and did you feel you needed to verify the output, or was just running it enough? Not trying to judge here! Just curious because nngroup just published a user study that showed only a tiny percentage of people actually cross-checking LLM-assistant output - not that it was wrong, they weren't checking either, just that most of the users in their study didn't feel it was necessary.)

It's something which probably needs to be done again -- I included a few parameters in the script that allow it to be used for a limited set of similar workflows in the future. I read over the script, didn't spend loads of time understanding it, but the logic looked roughly correct. I generally read the code the LLM generates, then test it. I also try to include dry run modes whenever possible, so I can validate the mutating behaviour is correct before actually running any mutating commands -- I'm far more likely to do this with LLM-generated scripts than my own code, I've found, though perhaps that's just laziness. :)

There are a ton of development tasks that I do infrequently enough that I have to google how to do them every time, because I forget how to do them.

For those things, I can just ask Chat-GPT to write the first draft of it, and it saves me about 80% of the time. I always have to end up doing a few edits, but it works out.

Also dropping in an indecipherable page of logs and immediately getting the source of an error with at least a suggestion of a direction is really useful.

It seems like it can be transformative organizationally at some point, but for individuals I agree I don't see what people are getting. I seem to be able to pretty much do the things I want to do and achieve the things I want to achieve in terms of sheer information discovery and generation. The qualities that might improve my life are things like better relationships with my family, finally having a child and ensuring it's healthy and happy, my wife somehow having her alcoholism cured, possibly overcoming certain physical limitations related to aging and injuries. I don't feel like any meaningful life outcomes are currently being impeded by an inability to generate code fast enough or find information I need from the Internet or the fact that my queries and commands to computing systems largely have to follow a pre-defined structure.

Legitimate advances in cheap robotics, on the other hand, could maybe make a dent. It feels to me like the things that might make a difference in my life are help with physical labor, not intellectual labor. A nanny, a maid, a driver, that I don't need to pay a salary to.

It is to a search engine what a search engine is to physical libraries.

same we had chatbots and voice search for years. Even currently there was the lawyer and open ai made up complete nonsense and that doesn't appear unique to just that instance. We're still required to validate information.

I have used it to help write some documents and get ideas, but I feel like there should be more than help writing some text.

Given some of the applications I've seen it used for (write and respond to emails; write code for you, and not just boilerplate), I'm more concerned about LLMs dumbing us down over time.

This is essentially what Windows Copilot seems to be attempting to do currently. There's obviously downsides to this approach from a security concern - it doesn't live on device, it's on the cloud.

I actually think Apple is incredibly positioned here. All of their devices have dedicated NPUs, if they can optimize an LLM to run locally on these in a way that's performant/accurate, the power could be incredible for a local AI that helps out. It'd likely be a huge sales driver as well - Pro devices might have faster/more accurate AI assistants. If it's purely local, the access to personal data becomes less of a concern. Super curious to see how this evolves.

Apple's first foray into this kind of tech would have been siri, but they made the baffling decision to send siri requests to their servers and back rather than running locally. I know at some point that changed a while back, can't remember when, but siri is still so bad at doing/understanding very basic things that my confidence in apple is not super high to wield this tech.

> I actually think Apple is incredibly positioned here.

Does Apple even have an AI story? Siri is still the worst of the lot for voice. I don't think they are doing anything in the LLM space so....?

There's all the ML features that have that Android and windows are still catching up to. Let me know when I can search for that paper document I took a photo of by ctrl+f for a string I remember in it.

To do what though? Even the linked thread has no concrete examples.

If I had a human personal assistant, most (all?) the value to me would be doing physical errands. I'm not so busy or important that I'd need someone to manage my calendar or correspondence. Travel planning could be helpful, but I don't know if the "personal" aspect of it has that much incremental value over a centralized model...

The personal aspect would be connections to your contacts, apps and devices (if you have that kind of thing). If it was smart/reliable enough it could book appointments, order dinner, plan your weekend based on events people pinged you about.. that kind of thing.

That is, if you trust it enough to plug into your personal ecosystem and spend your money..

I’d assume anyone living such a busy life he needs a personal assistant to make dinner appointment decisions for him, most likely already has one.

One of the most basic tricks in product management, when validating a product idea, is to ask your potential customer “how do you currently try solve this problem?”. Because oftentimes these problems are fantasies nobody actually cares enough to solve.

> If it was smart/reliable enough it could book appointments, order dinner, plan your weekend based on events people pinged you about.. that kind of thing.

The way I see it, if I had an ever present human assistant following me around 24/7, I would definitely get some value from being able to offload parts of those tasks. However my gut says that the amount is quite small, since I would still need to direct them to do stuff for me, and confirm everything. Most of the value I've seen from EAs is helping to coordinate competing priorities, and I just don't have that many.

On top of that, a lot of the little ways that people get value from assistants (calling up hard to reach restaurants, reminding you to send cards and gifts) lose it's value if EVERYONE has access to an assistant.

I've been thinking about how these personal LLMs fit into Klara and The Sun[0], book by Ishiguro from a few years ago. Written in first person from someone named Klara, who is an AF (artificial friend) "human" who is an assistants for families and children.

The scene in the book that stuck out to me, and that I feel fits with personal AI / LLM, was the scene where a person in the AF store was looking to possibly buy an upgraded AF. The one the family had was something like 3.5, and now that 4 was around, they should ditch the outdated AF and get a new one with more learning power.

If I was looking to get into the personal LLM space, and I agree with the reddit post linked, I feel like the process will be to be able to quickly spin these up and train them, and be aware that new ones are going to come and don't be overly attached to the old. Weird to type that out.

[0]https://en.wikipedia.org/wiki/Klara_and_the_Sun

Surely if you got a new model you would want to feed it back up with all you had inputted in the last one? What else would there to be attached to? Something like its "personality" is necessarily just fleeting effect of a given prompt, right?

Maybe the year of the LLLM has finally arrived? LLLM, as in: the Linux Large Language Model [enabled Desktop]. sudo apt install llm, use it with any app, you get the drift...

This is not about hype, AGI, singularity or gazillions. Its about enabling individuals to get on top of the information fire-hose with the help of an important data processing tool. In a private and empowering manner. Nothing more, nothing less.

Sure there are certain hardware and data issues (where / how to train the models, with what data, how to ensure speedy inference in interactive use etc). But those are tangible targets people can chew on. No obvious show-stopper.

Is the majority of current LLM research reaching for the mythical grail of AGI? Or is there enough research effort being poured into how to retrain networks efficiently and how to finetune accurately on small amounts of data?

My personal knowledge base and personal correspondence is miniscule in comparison to the great swaths of the internet these things are trained on. Vague, inaccurate or non-comittal answers based on my personal data are perfectly well served by my magic 8 ball at the moment.

My first thought after ChatGPTs launch last year was that the eventual goal is a personal AI. I'm very forgetful, having an AI assistant would be immensely helpful.

Can go further than that of course, I like to cycle and weight lift, I can see a future where I have an AI let me know where I should push and conserve energy on a bike ride/develop a training plan for me.

I used https://trainasone.com/ for a bit when I used to run more. Seems like a perfect use for AI.

There are already a bunch of tools that do that sort of thing (even Strava has it these days), I'm not sure how much LLM-based tech would help.

Personal LLMs make a lot of sense. Until they don't.

Right now, having a OCR combined with an LLM would open the way to a personalized assistant (rather a well working search machine and letter/email generator). No need to manually manage or catalogue documents, photos, videos, contacts, it would all be available by grepping your input stream smartly.

But once everyone uses such a tool, why bother generating human-readable text in the first place? There's so much efficiency to be gained: Send plaintext instead of html or pdf. Replace plaintext with its embedding. The sending AI could learn what the receiver understands. Finally, don't send stuff at all. Just point to the relevant vector DB entry.

In the end, the LLM itself will be obsolete, except as an appendix to communicate with humans. For all other tasks it will be superseded by a evolvednit engineered, form of machine-to-machine language.

I am adjacent to personal LLMs at work. They've been trying to make them work for a number of years.

LLMs make it simpler to query the data, but its not quite there yet for useful work.

For example if you have an assistant and you talk off and on about booking for $band, they will know that you like music, and that genre. They can alter the context of replies to suit.

With LLMs, the link between shortterm and long term recall is broken.

Now, this might be partially solvable by having an intermediary LLM that search the interaction database for context and feeds it into the main interaction LLM. But I'm not convinced that its a long term or viable solution. the LLM is not building a personal insight into your responses, its just having one shot context injected.

The only way that would be possible is if an llm were in a mode of always being trained and could count on being accurate. Which i dont think is possible? So trying to use the calculated weights as a massive compressed data source that knows everything is a mistake and is just leaning into the hype. which this is doing imo.

The only thing it seems to enable is using natural language as an interface, instead of a programmed api. Text goes in, translates to something the computer understands, text comes back to the use. Instead of like rpc and and error code. And there is code auto complete.

Are there any open source projects that make this possible today? Something you can just throw data at (in any format) and build up a training library?

Yeah sure, some time in the future. For now I think the fuzzyness and mediocre quality of the responses you get at scale are probably big showstoppers for usage at large.

Also, I've not followed this domain over the summer, are we still at 50k token caps on the state of the art models?

Claude has been able to do 100k for months and is as good, if not better (IMHO) than GPT4 for some things.

Okay, 100k means we are getting to the point where one can safely edit a large piece of text, say a blogpost or a small scientific paper if you are lucky. It still means most of the long running interactions will get clipped/compressed relatively soon.

But good to see the industry evolving

I recently switched to Claude after using ChatGPT for several months. CG tells you why something is great, while Claude tells you how it's useful. CG is more of a psychoanalysis chatbot, which would be good as a virtual partner.

We're at that part of the Gartner hype cycle.

Reply on news.ycombinator.com