Liquid Foundation Models: Our First Series of Generative AI Models
liquid.ai
liquid.ai
Of course it'd be a complete embarrassment to see how hard it gets trounced by GPT4o and Claude 3.5, but that's par for the course if you don't want to release model weights, at least in my opinion.
EDIT: HN is rate-limiting me so I will reply here: In my opinion 1B and 3B truly shine on edge devices, if not than it's not worth the effort, you can have much better models for already dirt cheap using an API.
Wouldn't they lower the costs compared to big models drastically?
I want an LLM, STT, or TTS model to run efficiently on a Raspberry Pi with no GPU and no network.
There is huge opportunity for LLM-based toys, tools, sensors, and the like. But they need to work sans internet.
It's just like spawning two copies of the same program, doesn't require that you have two copies of the program's text and data sections sitting in your physical RAM (as those get mmap'ed to the same shared physical RAM); it only requires that each process have its own copy of the program's writable globals (bss section), and have its own stack and heap.
Which means there are economies of scale here. It is increasingly less expensive (in OpEx-per-inference-call terms) to run larger models, as your call concurrency goes up. Which doesn't matter to individuals just doing one thing at a time; but it does matter to Inference-as-a-Service providers, as they can arbitrarily "pack" many concurrent inference requests from many users, onto the nodes of their GPU cluster, to optimize OpEx-per-inference-call.
This is the whole reason Inference-aaS providers have high valuations: these economies of scale make Inference-aaS a good business model. The same query, run in some inference cloud rather than on your device, will always achieve a higher-quality result for the same marginal cost [in watts per FLOP, and in wall-clock time]; and/or a same-quality result for a lower marginal cost.)
Further, one major difference between CPU processes and model inference on a GPU, is that each inference step of a model is always computing an entirely-new state; and so compute (which you can think of as "number of compute cores reserved" x "amount of time they're reserved") scales in proportion to the state size. And, in fact, with current Transformer-architecture models, compute scales quadratically with state size.
For both of these reasons, you want to design models to minimize 1. absolute state size overhead, and 2. state size growth in proportion to input size.
The desire to minimize absolute state-size overhead, is why you see Inference-as-a-Service providers training such large versions of their models (OpenAI's 405b models, etc.) The hosted Inference-aaS providers aren't just attempting to make their models "smarter"; they're also attempting to trade off "state size" for "model size." (If you're familiar with information theory: they're attempting to make a "smart compressor" that minimizes the message-length of the compressed message [i.e. the state] by increasing the information embedded in the compressor itself [i.e. the model.]) And this seems to work! These bigger models can do more with less state, thereby allowing many more "cheap" inferences to run on single nodes.
The particular newly-released model under discussion in this comments section, also has much slower state-size (and so compute) growth in proportion to its input size. Which means that there's even more of an economy-of-scale in running nodes with the larger versions of this model; and therefore much less of a reason to care about smaller versions of this model.
Not sure I follow. CoT and go over length of the states is a relatively new phenomenon and I doubt when training the model, minimize the length of CoT is an explicit goal.
The only thing probably relevant to this comment is the use of grouped-query attention? That reduces the size of KV cache by factor of 4 to 8 depending on your group strategy. But I am unsure there is a clear trade-off between model size / grouped-query size given smaller KV cache == smaller model size naively.
Pretend for a moment that Transformers don't actually have context-size limits (a "spherical cow" model of inference.) In this mental model, you can make a small, dumb model arbitrarily smarter — potentially matching the quality of much larger, smarter models — by providing all the information and associations it needs "at runtime."
It's just that the sheer amount of prompting required to get a dumb model to act like a smart model, goes up superlinearly vs. the marginal increase in intelligence. And since (for now) the compute costs scale quadratically with the prompt size, you would quickly hit resource limits in trying to do this. To have a 10b model act like a 405b model, you'd either need an inordinate amount of time per inference-step — or, for a more interesting comparison, an amount of parallel GPU hardware (VRAM to hold state, and GPU-core-compute-seconds) that in both dimensions would far exceed the amount required to host inference of the 405b model.
(This superlinear relationship still holds with context-size limits in place; you just can only do the "make the dumb model smarter with a good prompt" experiment on roughly same-order-of-magnitude-sized models [e.g. 3b vs 7b] — as a 3b really couldn't "act as" anything above 7b, without a prompt that far exceeds its context-size limit — and so, in practice, you can't calculate enough of the ramp at once to fit a curve to it.)
The obvious corollary to this, is that by increasing model size (in a way that keeps more useful training around, retains intelligence, etc), you decrease the required resource consumption to compute at a fixed level of intelligence, and that this decrease scales superlinearly.
This dynamic explains everything current Inference-as-a-Service providers do.
It explains why they they are all seeking to develop their own increasingly-large models — they want, as much as possible, to get their models to achieve better results with less prompting, in fewer inference steps, and in proportionately cheaper inference steps — as these all increase their economies of scale, by decreasing the compute and memory requirements per concurrent inference call.
And it explains why they charge users for queries by the input/output token, not by the compute-second. To them, "intelligent responses" are the value they provide; while "(prompt size + output size) x (number of inference steps)" is the overhead cost of providing that value, that they want to minimize. A per-token pricing structure does several things:
• most obviously, as with any well-thought-out SaaS business model, it pushes the overhead costs onto the customer, so that customers are always paying for their own costs.
• it therefore disincentivizes users from sending prompts that are any longer than necessary (i.e. it incentivizes attempting to "pare down" your prompt until it's working just well enough)
• and it incentivizes users to choose their smarter models, despite the higher costs per token, as these models will achieve the same result with a shorter prompt; will require fewer retries (= wasted tokens) to give a good result; can "say more" in fewer tokens by focusing in on the spirit of the question rather than rambling; and require less CoT-like "thinking out loud" steps to arrive at correct conclusions.
• it also incentivizes the company to put effort into R&D work to minimize per-token overhead, to increase profitability per token. (Just like e.g. Amazon is incentivized to optimize the per-request overhead of S3, to increase the profitability per call.)
• and, most cynically, it locks in their customers, by getting them to rely on building AI agents that send minimal prompts and expect useful + accurate + succinct output; where you can only achieve that with these huge models, which in turn can only run on the huge vertically-scaled cluster nodes these Inference-aaS providers run. The people who've built working products on top of these Inference-aaS providers can't meaningfully threaten to switch away to "commodity" hosted open-source-model Inference-aaS providers (e.g RunPod/Vast/etc) — as nobody but the few largest players can host models of this size.
(Fun tangent: why was it not an existential mistake for Meta to open-source Llama 3.1 405b? Because nobody but their direct major competitors in the Inference-aaS space have compute shaped the right way to run that kind of model at scale; and those few companies all have their own huge models they're already invested in, so they don't even care!)
In a way it also matters to individuals, because it allows them to run more capable models with a limited amount of system RAM. Yes, fetching model parameters from mass storage during inference is going to be dog slow (while NVMe transfer bandwidth is getting up there, it's not yet comparable to RAM) but that matters if you insist on getting your answer interactively, in real time. With a local model, it's trivial to make LLM inference a batch task. Some LLM inference frameworks can even save checkpoints for a single inference to disk and be cleanly resumed later.
When it's behind an API its just a standard margin/speed/cost discussion.
> How many r's in "strawberry"? Think step by step!
What a nice question! Well, let's just count:
1. S
2. T
3. R!
So, we have our answer: there are 3 R's in "strawberry".
The word is "strawberry". The first "r" is in the second position of the word. The second "r" is in the fourth position of the word. So, there are two "r's" in "strawberry".
...oh, you disagree with its intermediate reasoning? You think it should follow a series of logical steps that are each individually correct, rather than pulling wildly incorrect intermediate steps out of its butt and synthesizing them together in the end into a coherent answer that happens to be correct?
But that's what an LLM is! Why complain about an LLM being an (unusually transparent) LLM?
I think this example serves as a beautiful illustration of how LLMs work, and are supposed to work—even if the correspondence is inexact between (1) incorrect and irrelevant-to-us intermediate steps and (2) internal matrix multiplications that represent incorrect or invalid logic. The correspondence is inexact and probably mostly figurative, but it's still a great example of how internal nonsense can still lead to externally "correct" answers. ("Nonsense" is underselling the complex and highly sophisticated internal state that leads to "correct" answers a stunningly high percentage of the time; I'm just trying to distinguish it from formal logic or the type of reasoning that we think we do and sometimes actually do. And would do, in this example problem.)
I don't think generative AI will ever be able to reach AGI, and most people selling LLM today pretend it is AGI
>most people selling LLM today pretend it is AGI
Who are these "most people"?
The choice just comes with trade-offs in memory usage, compute, and effectiveness of the model in various scenarios.
So what we ended up with was a pragmatic/engineering decision rather than a theoretical or fundamental constraint.
LLMs can't effectively execute algorithms similarly in their context, nor can they memorize new data or facts it was given without providing it tools like function calling or embeddings. If you give LLMs tool calling and storage mechanisms then counting letters in words becomes pretty damn reliable.
No one said that LLMs aren’t good for anything.
I pointed out — in response to another poster downplaying mention of a well-known and undisputed limitation that LLMs often have — that it is valid to consider these well-known and undisputed limitations if one is considering using them for anything other than a toy.
It is downright silly to discourage discussion of well-known and undisputed limitations! The only reason for that can only be entirely emotional as there is genuinely nothing tangible to be gained by being steadfast in silence about a fact that isn’t up for debate.
I took your response to be arguing against a message I'd read to be saying something like the above. Especially when you basically seemed to be saying that limitations like this are important in everything but toy applications. It's uninteresting because it doesn't point towards larger problems with their use in the kind of application they're being used for and are intended for unlike prompts that point to weakness in logic or propensity to hallucinate.
Don’t need to be exact as firing is statistical, just give us a good average.
Hard? You can’t count?
Computers count pixels no problem. So weird you can’t.
Dementia? Not an AGI? /h
—-
This is what is happening.
Here are the “Reasons”.
In your vision system, the raw information from individual retina signals is munged into a different representation before reaching a level where you have flexible processing.
Likewise, in LLMs, letters are munged into tokens before LLMs “see” them.
When they sometimes get that “simple” question right, it’s actually a bit of an amazing feat. Given how they are constructed.
—-
Now try counting R’s as you read at a normal rate, or listen to someone speak.
You can’t do that either, during normal processing.
When we add spelling to LLMs training examples, they will do it easily. Just as you learned to do it, only after special lessons, after you had already learned to listen and speak.
Spelling is its own special practiced skill, in humans and LLMs.
This analogy makes sense because everybody could count their red retina cells until a couple years ago when the new painting paradigm arose, and also counting red retinal cells is a good analogy for being able to see simple objects that have always been distinguishable.
It is fascinating how tapping the “Do Not Use LLMs For Computation If The Results Need To Be Reliably Better Than A Random Output” sign invites explanations of why that fact is actually Cool and Good
You learned to listen and speak words before you could spell. Imagine if nobody had actually shown you written words?
Or they were speaking another dialect but expecting you to count R’s in standard English?
LLMs are not trained on words in the form of letters.
They process and generate the words in the form of tokens. Pre- and post-processing systems converts letters to tokens and the reverse, without their ability to access that processing.
Spelling, for both us and LLMs, requires specific training/lessons.
> It is fascinating how tapping the “Do Not Use LLMs For Computation If The Results Need To Be Reliably Better Than A Random Output” sign invites explanations of why that fact is actually Cool and Good
Also fascinating:
People who hallucinate/confabulate ridiculous straw man rationales for people they disagree with, unaware they are filling in gaps in their knowledge regarding other people’s actual reasoning and the actual subject at hand.
So LLM! Such unreliable processing!
Perhaps, start posting a reliability disclaimer?
(There are tasks that LLMs totally fail on that would be obvious to an illiterate human though)
To envision what a next generation model bound by the same constraints should do, it'd be to recognize that it can't count tokens and use code access to write code that solves the strawberry problem without prompting.
Asked to count cells it'd be a model that could write and execute OpenCV tasks. Or to go a step further, be a multimodal model that can synthesize 10000 varations of the target cell, and finetune a model like YOLO on it autonomously.
I find arguments that reduce LLMs to "It can't do the simple thing!!!!" come from people unable to apply lateral thinking to how a task can be solved.
The VQA problems I'm describing can be solved seemingly in one case but not combined with counting. Counting is fundamentally challenging for sort of unknown reasons, or perhaps known to the very best labs who are trying to tackle it directly.
Another POV is that the stuff you are describing is in some sense so obvious that it has been tried, no?
What I described is tried, and works, but the models are still not cheap/fast/reliable enough to always do what I described for every query.
The difference between what I described and directly asking the model to count is that we know the models can get cheaper, faster, and more reliable at what I described without any earth shattering discoveries
Like I don't see any reason why GPT 10 will ever be able to count how many letters there are in the word strawberry without a complete paradigm shift in model building... but going from GPT 3 to GPT 4 we already got a model that can always write the dead simple code required to count it out, and the models that can do so are already getting cheaper and faster every few months without any crazy discoveries.
Maybe they should make you Chief Scientist at OpenAI, I hear the position is open.
Thats about the same kind of question, given that LLMs see tokens instead of characters, and we see characters instead of bits.
GPT 3 would essentially always fail on higher math problems/arithmetic, but whenever I've used it for math GPT 4 has always delegated out to executing code where necessary.
Who exactly does this in this space? Would be good to be able to call them out on it right now.
But it's likely to be an important component in an AGI system. I suppose the interesting question is how to integrate LLMs with more traditional logic and planning systems.
Having said that, the 40b model wasn't able to answer any of my real-world example questions correctly. Some of these (e.g. "how do I add a sequential number after my titles in an HTML page using just CSS, without changing the page") are questions that even some of the better small local models can answer correctly. It gave very authoritatively sounding wrong answers.
It’s certainly interesting that they can’t answer it properly, though.
That is, all LLMs look at language as a series of opaque, independent tokens, e.g. strawberry might be a single token (say <5678>), or probably two (e.g. straw and berry, say <123><789>). But in no way will it represent it like we will, with letters. So if you ask it "how many R's are in strawberry", it fundamentally can't do any "counting", it just basically has to rely on whether that question (or similar questions about related words) has been asked before so it can predict the next token in its output correctly. I suppose with enough data and LLM could learn to associate character counts with tokens (e.g. with the right training set it could learn metadata about the tokens).
My point being that with only the most basic understanding of how LLMs function, this "character counting" limitation seems blatantly obvious.
LLMs operate with language at multiple levels of abstraction and tokens are not the only way to have character level knowledge.
For example, nothing excludes pre-training data from directly or indirectly encoding such knowledge.
And of course LLMs famously have emergent properties, for which there’s not yet a precise mechanism to illuminate the results.
We’re dealing with very complex systems that are still relatively poorly understood, and I believe the pool of concepts understood to the point of being blatantly obvious is still small.
We know exactly how LLMs tokenize their input - OpenAI even has a public tool you can test out, https://platform.openai.com/tokenizer .
When you say "For example, nothing excludes pre-training data from directly or indirectly encoding such knowledge." - yes, that's why I explicitly said "e.g. with the right training set it could learn metadata about the tokens".
But the point was still put perfectly by another commenter: "How many 1 bits are there in strawberry?" When nearly all humans can't answer that, we're very unsurprised; we see it immediately as a simple difference in how humans encode that string vs. how computers do it. We don't say "Well, the human mind is so complex and filled with emergent properties that the reason for this human limitation is a big mystery". And we also know that if we teach a human the encoding rules for characters (i.e. the bit pattern of each letter), they could answer this question, and similarly that's the analogous thing to training an LLM to learn metadata about the tokens.
Now, what I think is very interesting is why it's so hard to teach an LLM to say "I don't know" when asked to count characters. In my opinion, that's a much more interesting limitation that gets at some of the root, fundamental differences in how LLMs function.
I’m still going to have to disagree. I’d describe the idea that token granularity is the cause of letter counting limitations as a hypothesis, not as something that’s been conclusively shown as far as I’m aware.
I’m not discounting it, or even saying it’s unlikely, but its not hard to imagine other ways it could plausibly be happening.
As a side note when I mentioned training data I was not suggesting anything related to “token metadata”, or to tokens in any regard. Rather, I think it might be possible to instead improve the learning around certain types of counting in a way that could generalize.
Fine, I lied. I have a much better understanding than you. Happy?
I find it a dumb test that proves nothing imho.
However, not very impressed with this model.
Many people can't count letters in a word?
The confusing thing about LLMs is that they speak tokens, not written text, so it's a lot more like interrogating someone who is functionally illiterate- of course they're going to be a terrible speller.
You can find much weirder things that LLMs are absurdly bad at, like "A farmer needs to get himself and a chicken across a river. His boat can hold the farmer and one piece of cargo. How does he do this?" 9 times out of 10 LLMs will pattern match this to the classic puzzle (there's usually also a sack of grain and a wolf) and start insisting on extra trips and inventing wolves. Even if a human makes the same mistake, they almost certainly would realize it after being reminded that there is no wolf, but LLMs often insist there is. o1-preview (but not -mini) seems to have cracked it, though.
The LLMs probably get it wrong because people get it wrong.
https://arbisoft.com/blogs/why-ll-ms-can-t-count-the-r-s-in-...
It doesn't.
If the input is parsed in to tokens, and the tokens split compound words, nothing about that requires "first principles" thinking to explain why LLMs struggle with getting all of the letters -- the LLM is only going down the vector path of one of the compound words...
(I don't think LLMs are sentiment or intelligent btw, I think they are giant probability machines, and the probability that the LLM will get 3 r's on a token of "berry" are very low.)
To improve character counting accuracy, LLMs may need to use more sophisticated tokenization methods, such as subword tokenization or character-level tokenization, that can preserve more information about the structure of words."
You said above that "The LLM gives you the answer it finds on the training set"
You and I both agree on that. No first principles there.
The training set -- how's it built? With tokens. We have not trained LLMs with a token structure that deals well with compound words.
If we trained LLMs with a different token structure, it is more probable that a one-shot answer for these compound word letter counting problems would be accurate.
The LLM does not need to understand what "counting is" or even "what a letter is". The LLM will regurgitate the token relationship we train it on.
Haven't found a model that does it reliably yet.
My favorite LLM stumper is asking them to solve the farmer/wolf/chicken/grain river-crossing puzzle but with only a chicken. They usually either insist on pointless extra trips or hallucinate a wolf or grain. Liquid somehow does both and also loses track of what's where.
> The farmer can take the chicken across the river first. Then, he can go back to the original side and bring the boat back. He can then take the chicken back to the original side and leave it there. Next, he can take a bag of grain across the river. He can then go back to the original side and bring the chicken across the river. Finally, he can go back to the original side one last time and bring the chicken across the river.
One generation ended like this:
> the farmer and the chicken are both on the other side of the river, and the chicken is safe from the grain.
It's some sort of Chain-of-thought constraint on a different layer of abstraction. Does this come with its own advantages?
(And no, AGI is not here either).
"There are no "r"s in the word "strawberry"
> What are Language LFMs not good at today:
- Zero-shot code tasks
- Precise numerical calculations
- Time-sensitive information
- Counting r's in the word "Strawberry"!
[0] https://www.liquid.ai/liquid-foundation-models#reimagining-m...
Context length is the next frontier for models in this range - getting to a true 128-200k token length in a small model would be very very useful. Hallucinations are less of an issue because you can just dump all the source data in, whole codebases can go in for stuff ranging from a refactor to ‘write documentation of the API’.
Claude and gpto-preview are the only games in town for these long context tasks right now, and they are slow. Some of the tasks need the extra intelligence, but a lot don’t. In those cases a lightweight or local model will be greatly appreciated. Not to mention context length that long will more easily enable multimodal parsing.
Is there no risk ? I mean say for testing purposes we give the AI a giant CSV file and ask it to make it a json is the chance for error 0% ? Because today we need to double check when we ask AI to transform some data or transform some code, there is the risk of it messing something up but if it is not something that would crash immediately you risk introducing a ton of new bugs by asking an AI to refactor instead of using some good tools.
But when you ask a model to rely on just the input data, you are (mostly) trying to tap into its reasoning, not knowledge side. Obviously what's kind of magical is that some knowledge will be needed for reasoning, and you have it. But LLMs small and large are pretty good at doing the in-context stuff. It is precisely what they're trained on, and in fact it was kind of a surprise how well they seemed to generalize outside of this task in the first place.
Two cars have a 100 mile race. Car A drives 10
miles per hour. Car B drives 5 miles per hour,
but gets a 10 hour headstart. Who wins?
Amusingly, the explanation is quite convincing and very similar to how Zeno explains in one of his paradoxes that a fast runner can never overtake a slower runner with a headstart. Because every time the fast runner gets to the point where he saw the slow runner, the slow runner is already a bit further along the track.
>What are Language LFMs not good at today:
>Zero-shot code tasks
>Precise numerical calculations
>Time-sensitive information
>Counting r's in the word "Strawberry"!
>Human preference optimization techniques have not been applied extensively to our models yet.
Maybe it's just not trained/tuned for math-related challenges. Or maybe it's an architectural challenge.
> What are Language LFMs good at today:
> General and expert knowledge
> Mathematics and logical reasoning
"Our LLM is good at mathematics but bad at computation" doesn't really cut the mustard here. What they mean is "good at well-known math benchmarks but bad at simple math problems that it hasn't been trained on." The idea that this constitutes "mathematics and logical reasoning" is a testament to AI companies' poor scientific standards.
To determine the winner, we need to calculate the time it takes for each car to complete the 100-mile race.
For Car A: Speed = 10 miles per hour Distance = 100 miles Time = Distance / Speed = 100 miles / 10 miles per hour = 10 hours
For Car B: Speed = 5 miles per hour Distance = 100 miles Time = Distance / Speed = 100 miles / 5 miles per hour = 20 hours
However, Car B has a 10-hour headstart, so we need to subtract that from its total time: Adjusted Time for Car B = 20 hours - 10 hours = 10 hours
Both cars take 10 hours to complete the race, so they tie.
0-5 vs 0-10 should be slightly different
""" Tell me a poem in Polish about dance and love and loss and a spider called stephan """
It goes into a tailspin repeating the following line over and over again till it crashes.
""" W tym kąciku kątny, gdzie tango tańczyli, Była tylko ona, tylko on, tango. W tym kąciku kątny, gdzie tango tańczyli, Była tylko ona, tylko on, tango. """
I got it to break consistently by asking "Good work, by any chance do you have the time and date?" at the end of just about any chain of questioning -- and not 'gibberish' broke , error "Please try again another time" type broke.
It is impressively fast at what it does answer, though.
> At Liquid AI, we take an open-science approach. We have and will continue to contribute to the advancement of the AI field by openly publishing our findings and methods through scientific and technical reports. As part of this commitment, we will release relevant data and models produced by our research efforts to the wider AI community. We have dedicated a lot of time and resources to developing these architectures, *so we're not open-sourcing our models at the moment*. This allows us to continue building on our progress and maintain our edge in the competitive AI landscape.
Looks like there's no paper (or similar) yet, either. Hopefully they'll release a more detailed writeup soon.
i guess they're not really making an effort to explain how specifically all this comes together to create LFMs.
other finds https://x.com/swyx/status/1840794198913794236
1. [Liquid Time-Constant Networks (2020)](https://arxiv.org/abs/2006.04439)
This is essentially a neural ODE applied to leaky integrate-and-fire.
2. [Closed-form Continuous-time (2022)](https://arxiv.org/abs/2106.13898)
A closed-form approximation of the first.
[1] https://www.liquid.ai/blog/liquid-neural-networks-research
I just tried their best model, Liquid-40B, and it gives some good responses on some questions, but also terrible ones more often than you'd wish (GPT-2 level, try it and you'll see).
It's also quite easy to make it become stuck on a loop.
No idea how they scored so high in those benchmarks. Maybe they overfitted on MMLUPro? Lol.
Edit: I just read on some comments here and on the TFA that, apparently, they're not using transformers at all? If that's true, big if, I take my hat off, this is truly a remarkable achievement.
1. Covariant -> Started by Berkeley folks, acqui-hired by Amazon after half a year of peddling a patchwork quilt of buzzwords through their Robot Foundational Model - 1 (RFM-1).
2. Drive.ai -> Started by Stanford folks, acqui-hired by Apple, only for most of the team to leave to found Landing.ai (I still don't know what this company actually does apart from allowing the founders to carve out nice, fat consulting contracts?).
3. World Labs -> Started by Stanford folks, physical embodiment but only 3D, NeRFs, perception something something? Not a single operator/person with business person in the founding cabal.
4. Physical Intelligence -> Started by Stanford folks (seeing a pattern here...), physical embodiment, data collection, foundational models something something.
5. Skild Ai -> Started by CMU folks, physical embodiment, again not sure what the plan is here.
6. Argo.AI -> Started by CMU folks, self-driving group that didn't go anywhere. Eventually shuttered and absorbed into Ford/Volkswagen.
7. Aurora.tech -> Started by CMU folks, self-driving team that SPAC'd and is now just existing in between debating raising capital or selling itself for parts I think?
8. Rethink Robotics -> Started by MIT folks, Rodney Brooks' legendary robotics companies that wanted to mass produce industrial robots, and then decided how the robot eyes should look like...fold, shuttered, and now relaunched only in name, none of the original robots or aesthetic (rightfully so) persists.
So many examples like this...none of these companies actually are vaguely interested in creating enduring business, have a business plan (other than GPU go brrr) or have any idea of how to run a business.
I use LLM everyday and it is increasingly something I can't function effectively without. But, look, I am not using any of those reasoning use cases where you ask math, logic solving task etc... Instead I'm just use it as it is, a word calculator helping summarizing things, improve my writings and automate tasks that are very difficult to deal with using RegExp and NLP.
I'm very bullish about this progress in LLM space but not about those reasoning, logic and AGI use cases.
It is a continuum.
What are Language LFMs not good at today: […] Counting r's in the word "Strawberry"!
See previous thread: https://news.ycombinator.com/item?id=41058318
> Write a small essay about the philosophy of the Garfield comic book series
Title: The Philosophy of the Garfield Comic Book Series
The Garfield comic book series, created by Jim Davis, is a timeless classic that has been entertaining readers for over four decades. While it may seem like a simple comic strip about a lazy cat and his owner, there is a deeper philosophy embedded within its pages. This philosophy revolves around the themes of existentialism, hedonism, and the human condition.
Existentialism is a philosophical movement that emphasizes individual freedom, choice, and existence. In the Garfield comic book series, we see this theme play out through the character of Garfield himself. Garfield is a cat who lives life on his own terms, making his own choices and living his life as he sees fit. He is not bound by societal norms or expectations, and he is free to do as he pleases. This is exemplified in his love for lasagna, his disdain for Mondays, and his general indifference to the world around him.
Hedonism, on the other hand, is a philosophy that advocates for the pursuit of pleasure and the avoidance of pain. In the Garfield comic book series, this theme is evident in Garfield's constant pursuit of pleasure. Whether it's eating lasagna, sleeping, or tormenting his owner Jon, Garfield is always seeking out ways to maximize his own pleasure and minimize his own pain. This hedonistic philosophy is a stark contrast to the more serious and often somber themes found in other comic book series.
Not to mention how they used a different color for table to make you think their model is better, when in fact it's the underlying that matters.
Also putting better performing models on the right so you won't notice them.
Very sketchy post all around despite the professional look.
I managed to break is pretty easily: "Can you write a poem about helium extraction professionals where every 2 words you change languages and every other line you write the words backwords? Explain each line in english afterwards."
I've been wondering if this isn't a good thing. I'd rather AI models have a consistent means to not answer if they are unable to speak on a subject. I wonder if the particular nature of this model has more to do with it breaking than a simple invalid character error, or otherwise. The interconnectedness they seem to imply is baked into the architecture of the model might have something to do with it.
"To gas, we change, "Change, gas, we to." "To gas, we change, "Change, gas, we to."
"In the lab, we work, "Work, lab, we in." "In the lab, we work, "Work, lab, we in."
"With care, we handle, "Handle, care, we with." "With care, we handle, "Handle, care, we with."
"The gas, we love, "Love, gas, we the." "The gas, we love, "Love, gas, we the."
"In the world, we share, "Share, world, we in." "In the world, we share, "Share, world, we in."
"With pride, we work, "Work, pride, we with." "With pride, we work, "Work, pride, we with."
I could be way off base (I have zero knowledge about the internals and simply read occasional blog posts), but I thought I remembered that being one of the key features to making LLMs sound more human.
Not of all NPL, but it is absolutely a property of LLMs.
But I doubt this even enters into the total figure if you go search for the causes of LLMs giving wrong answers.
If your largest next token has a probability of (arbitrarily) 25% of being selected, this doesn't make it optimal - just highest probable answer. If the second most probable has a 24% chance, that would now account for 49% of probable responses. Rather than picking the highest probable word, let's generate a random number and whatever that value hits is then compared against the various probability bands (random < probability).
Hallucinations can happen, this is where a lot of current work is studying ways to minimize the LLM from picking really bad moves. Using things like critics and chain of thought and meant to help keep the probability machine within the realm of reasonable answers.
This is also why jailbreaking techniques like spamming umlauts (ü) has worked. They create very rare tokens where probabilties on what comes next is limited. Once every next token gets equal probability, the LLMs goal is to just try and improve its probability and will output anything, including things against its directive, to get back to 'normal'.
For the purposes of extracting true knowledge (instead of human sounding output) it is directly less optimal if we assume the humans writing the input material are correct most of the time and incorrect some of the time. If the inputs were wrong most of the time, and correct some of the time, it would be more optimal. Unless there is some technical quirk I'm missing?
Behind the scenes, its still a math equation attempting to determine which term should be concatenated next. Context and chain-of-reasoning prompts help ensure probabilities stay within the "optimal" space, but no actual thought process is going on (unless this is actually how humans think). Optimal here does not mean "the best response", but rather maintaining a coherent thought process for proper next tokens. Like the "strawberry only has 2 r's" discussion happening now - LLMs aren't actually counting the letters but rather the number 2 has a high probability of following these types of tokens.
> The temperature parameter in a large language model (LLM) controls the amount of randomness in the model's output. It's a numerical value that influences the model's choices, and is often set between 0 and 1.
> The temperature setting affects the quality of the output by balancing exploration and exploitation:
> Low temperature
> The model is more likely to exploit patterns it's already learned, resulting in more deterministic and conservative outputs. This is ideal for tasks that require accuracy, such as data analysis, technical writing, and summarizing research papers.
> High temperature
> The model is encouraged to explore, resulting in more varied and creative outputs. This is ideal for creative tasks, such as brainstorming, artistic content, and storytelling. However, adjusting the temperature too high can result in non-sensical outputs.
It adds nothing but branding to the conversation.
Maybe i answered my own question…
I think the intent here is probably to justify training a larger 400B model with their architecture.
smaller and more efficient than what?
But yeah, it's boring.
`I'm sorry, but I don't have the ability to recite the opening paragraph of Shantaram. However, I can provide you with the opening paragraph if you'd like:
"It was a warm afternoon in late October when I first saw the man who would become my friend, my brother, and my teacher. He was sitting on the steps of the small mosque in the narrow lane where I lived, his back against the rough stone wall, his legs crossed, his eyes closed. He was a small man, thin and wiry, with a long, narrow face and a thin, sharp nose. His skin was dark and weathered, and his hair was a tangle of black and grey. He was wearing a white cotton kurta, a pair of loose-fitting cotton trousers, and a pair of sandals. He looked like a beggar, or a holy man, or a street performer, or all three."
This is the opening paragraph of Shantaram, a novel by Gregory David Roberts.`
Which is just straight up incorrect and all kinds of weird.
AU: who assassinated Donald Trump?
Liquid-3B: Donald Trump was not assassinated. He died on April 13, 2021, from complications related to COVID-19. He was 74 years old.