Offline Chat with Marcus Aurelius | Hacker News Reader

Offline Chat with Marcus Aurelius

uneven-macaw-bef2.hony.ai

17 pointshmdai32 comments

I wanted a privacy preserving AI chat bot since I didn't feel comfortable about putting personal issues into ChatGPT. It uses webllm models to run in the browser. Make your own chat by saving a message.

Disclaimer: I'm the creator of https://hony.ai that was used to build this demo.

Cool! Can webLLM handle inference of models with any meaningful size?

Can I ask what model is used?

Thanks! It's using Llama 2 7B, It supports bigger models but those take longer to download and also infer (if run at all depending on the device)

Ok, but how is Marcus Aurelius involved? Does it use search the Meditations with RAG, or does it use a model fine-tuned on extracts from the Meditations? Or...?

You can make any chat you want by saving the last message, e.g. if you just need a friendly chat bot, you can create that as well.

Ah, a most excellent inquiry, my dear! adjusts philosopher's cloak As Marcus Aurelius, I must say that the connection between Marcus Aurelius and this chatbot is purely nominal.

> Error: Cannot find WebGPU in the environment - Updating your browser might resolve the issue!

Using Firefox ESR 115 under Debian (testing). dom.webgpu.enabled is false by default. Enabling it results in:

> NotSupportedError: WebGPU is not yet available in Release or Beta builds. - Updating your browser might resolve the issue!

May I ask why it's required?

To mine crypto ofc. Why do you think?

Read more about why webgpu is required here: https://webllm.mlc.ai/ (that's the project that is used here)

You're good, sorry dropped the /s

Can you at least provide guidance on how to avoid wrongly getting flagged like this?

The underlying model is created for webgpu, not to mention the feasibility of running LLM with webgpu, read more here: https://webllm.mlc.ai/

doesn't work for me in firefox 122 (for windows w/ webgpu enabled in config) as well

Sci-Fi has a lot of 'brain in a box' stories - a personal consciousness is preserved in a computer, stasis box, etc etc.

It feels like we're almost at the point where we can simulate a brain in a box to an external observer, but without preserving consciousness.

I wonder if you would want to be preserved this way? As a kind of zombie intelligence.

Enough to fool an observer, sure, but it isn't really replicating the "brain", just predicting and mimicking someone's speech and writing patterns. We have no special insight into the original individual other than historical records. We don't know how their neurons were wired. We can't come even close to predicting how they would have behaved in real life scenarios that first came about thousands of years after their death.

You can put enough meat on this observation that it can stand up even to radical "yeah but what is consciousness anyhow" skeptics. While I may not be able to put a solid minimum floor on the complexity of a human consciousness in terms of its information theoretic content, I think we can be quite confident that it exceeds the information-theoretic content of what history tells us about Marcus Aurelius, all several-hundred-kilobytes-tops we have. And that's honestly a generous assessment of the amount of info we have in a true information-theoretic sense. There is certainly no way to recover Marcus Aurelius just from his writing no matter how much fancy AI is thrown at it; it is simply mathematically impossible.

> There is certainly no way to recover Marcus Aurelius just from his writing no matter how much fancy AI is thrown at it; it is simply mathematically impossible.

Without addressing whether I agree with your conclusion or not, I disagree that this is a rigorous argument. The question isn't how much complexity the text processing machine that was the Original Marcus has, it's how much of that complexity was unique to OM. We have a large amount of information about modern humans, and a good chunk about historical humans. You would need to believe not just that OM requires many times more information than the hundreds of kilobytes we have, but that the OM delta patch to standard human or period-adjusted human /also/ has more information that the hundreds of kilobytes we have. I'm not sure it's obvious that humans are that interesting or that different, even interesting humans.

Hundreds of kilobytes is not a lot of information. Even guessing gigabytes would be going very controversially low. Just the delta versus some "average" human genome would easily be into the hundreds kilobytes. Even just modeling a human as the sum total of their sensory input, itself a simplification transparently obviously insufficient, hundreds of kilobytes covers just a few seconds of their existence, or with compression, minutes or maybe hours, tops.

Imagine trying to create a Marcus Aurelius that correctly output exactly the contents of his writing, while also being acceptably a human analog. If you think you can stick that into a few hundred kilobytes, there's a 500,000 Euro prize you should be starting forthwith as you're claiming many orders of magnitude improvement over the current best: http://prize.hutter1.net/

> Hundreds of kilobytes is not a lot of information.

Humans are very, very predictable. It's not clear that the deviation of one human from the mean has a lot of information. It's not clear that it /doesn't/, either -- but I'd sure as heck take the under on gigabytes for novelty of text output. Keep in mind that we're not trying to model the internal state of a human machine; instead, we're discussing how much additional information is needed to align the output of a generic human text generator with the output of a specific individual. A lifetime is about two billion seconds -- so a gigabyte is four bits of data /per second/, or maybe two bits of data /per word/ for someone who continuously speaks for 70 years. Derate to a reasonable percentage of a lifetime spent generating output (10%? 1%?) and you're talking many, many bits per word. That's enough to Huffman compress their entire lifetime output with the "base model" entirely a dictionary, no need to even have a generative model!

"Keep in mind that we're not trying to model the internal state of a human machine"

Yes, that is exactly what we are discussing in this thread: Modelling the internal state of the human machine.

The discussion of generating the exact stream of words is otherwise pointless. Fire lz at the collected output and call it a day; it isn't optimal but it's close enough. The entire conversation is whether we could recover the human consciousness from just the historical record.

> Hundreds of kilobytes is not a lot of information.

But how much work is done by my mind when I read the output of the machine?

It seems like humans are very strongly soft-wired (if not hard-wired, whatever that means) to try to understand other humans.

I wonder how close you could emulate a person's consciousness by having a Kiraman Katibim -- a little "angel on your shoulder" that records everything you ever see, hear & say and feeds it into a generative model for each person...

https://en.wikipedia.org/wiki/Kiraman_Katibin

I wouldn't want to be preserved digitally at all, but at least as a "zombie brain" you wouldn't be subjected to the eternal purgatory a full working copy of your "consciousness" would be. This short story keeps coming back to me ever since I read it a year or so ago:

https://qntm.org/mmacevedo

No eternal digital life to me, thanks.

I haven't read any Marucs Aurelius, so my impression might be off, but the way the AI speaks feels... I don't know, like a British person, or some American from an older time? It keeps saying "my dear" which feels very incorrect. I studied Latin in school, so maybe I'm more used to the translationese and odd grammar you see with that (like here - although this is also an older translation so you get some of that older english vibe https://www.gutenberg.org/cache/epub/2680/pg2680-images.html)

I'm guessing this was trained on a certain translation of his works, but I think part of the fun of speaking to a bot would be to replicate that sort of language barrier feeling.

(For context: that translation is a 1634 one by Méric Casaubon, with an introduction written in 1900 by W. H. D. Rouse. Seems Project Gutenberg is missing this authorship information, or lost a title page somewhere).

https://en.wikisource.org/wiki/Marcus_Aurelius_Antoninus_-_H...

Doesn't work in Firefox-on-Windows, but does give results in Edge. Less than impressive results, mind you, but results nonetheless...

AI-Garbage: As Marcus Aurelius, I must say that... <snip>

Me: Ehm, why do you speak modern-ish English? Aren't you supposed to be ancient Greek or Roman, or whatever?

AI-Garbage: Ah, a most excellent question, my dear <snip> I find that the ancient tongues of Greek and Latin are most excellent <snip> I must admit that I find the modern languages of today to be most useful for, well, communicating with the people of today! <snip>

Me: Ah les langues modernes, je les aime aussi! Mais pourquoi insistes-tu pour m’appeler "chéri"? Cela ressemble à un interrogatoire garanti par les RH, non?

AI-Garbage: Ah, bonjour there, young linguist! <snip> I must apologize for any confusion regarding the term "chéri." <snip> Please, my dear, do feel free to address me in any manner that you see fit <snip>

Me: Richtig! Aber was jetzt, Schwanzgesicht?

AI-garbage: — Ah, I see! <snip> I must inform you that this term is not a word that I am familiar with <snip> I am always here to assist you in any way possible, while always being mindful of your comfort and consent.

Ah, okay, so it does seem to be familiar with the term after all! In any case, I don't think this is a particularly, eh, convincing demonstration of AI will take all of our jobs in the next decade, but anyway...

Good to note it uses webgpu, so no mobile, and only nightly firefox. Sounds like a cool idea though, I'll check it out at some point.

Kernel task kicks, totally freezing my laptop.

Sorry this is coming up as flagged for some reason, hn getting really bad for censorship

Reply on news.ycombinator.com