Open Challenges in LLM Research
huyenchip.com
huyenchip.com
I was interviewing with Claypot.ai and when I met her for my first conversation, she was on a walking treadmill and very clearly was more interested in a Slack conversation she was having.
She moved me on to the next round which I irrefutably bombed and was respectfully told that I wouldn't be moving on which was the right decision, but I'll never forget watching her walking motion while looking at Slack on her second monitor almost the entire time we were talking.
Looking back, it's a bit absurd to say that GOFAI would've got to AGI if only the Frame Problem could be solved. But the important point is why that sounds so absurd.
It doesn't sound absurd because we found out that the frame problem can't be solved; that's beside the point.
It also doesn't sound absurd because we found out that solving the frame problem isn't the key to GOFAI-based AGI. That's also beside the point.
It sounds absurd because the conjecture itself is... just funny. It's almost goofy, looking back, how people thought about AGI.
Hallucination is the Frame Problem of the 2023 AI Summer. Looking back from the other side of the next Winter, the whole thing will seem a bit goofy.
Model hallucinations seems to me like a fancy way to call the model results that make no sense (ie blatant errors). Plus it makes the model more humanoid.
The problem is we are so used to yielding to confidence we don't apply the necessary checks even when we know it is projected by machine.
Humans are certainly better but we don't have an absolute sense of what we do or don't know either.
Meanwhile, the neural nets (and ML) researchers just trucked on, with more compute power, and pretty much ignored any theoretical issues with uncertainty. And surprisingly, with lots of amazing results.
But now they hit the same wall, we don't actually understand how to do reasoning with uncertainty correctly. LLMs seem to solve this by "just mimic reasoning that humans do". Except because we lack a good theory of reasoning, it can't tell when mimicking is bad and when it's good, unless there is a lot of specific examples. So in the most egregious cases, we get hallucinations but have no clue how to avoid them.
LLMs transpose the problem by mimicing what humans would do
for most of us, what we wish for is what we believe.
It'll be interesting to see what improvements (in a lab or at a company) need to happen before most people use purpose-built LLMs (or behind the scenes LLM prompts) in the apps they use every day. The answer might be "no improvements" and we're just in the lag time before useful features can be built
But the biggest problem is that they take so much compute, which slows down both research and deployment. Only a handful of giant companies can train their own LLM, and it's a major undertaking even for them. Academic researchers and everyday tinkerers can only run inference on pretrained models.
I believe that better hardware architectures will have more impact on AI than better neural network architectures.
And I'd be able to train a learned optimizer to replace gradient descent as the training process.
Even without either of those, performance improves in a predictable way with more compute thanks to scaling laws.
You're correct that our architecture isn't adequate and biggest achievements lie there. I/O is not the problem, in fact, we have faster I/O. Because our I/O is dumb. We can place massive amounts of data in a linear memory buffer. But brains use massively associated memory structures. I/O of a network packet is easy. Associating that packet with preexisting context (such as TCP connection) is not that easy, requires structures, algorithms, memory locality, threading correctness, and procedural computing steps, because we abstract the context over a series of flat data.
If you're working on a subject hard, just a random flying info about something else that concerns you might trigger "I can't think about that right now" reaction in your brain, but the information has been digested. The packet has reached the adequate layer 7 ingress buffer just like that, but you don't want to context switch to the respective application intentionally.
There is also an elephant in the room and that is the native language, which shapes the way we think and process information. Imagine a CPU receiving an automatic microcode update the same moment when you as a programmer defined an abstract TCP stack in C or assembler, so it can optimize itself to the point of being able to switching to "thinking in TCP" mode.
So whoever is working on this problem, good luck because you have you have a lot of work to do to get Markov chains to only output facts and not just correlations of the training data.
ChatGPT is often 'confidently wrong' - I'm pretty sure I've been confidently wrong a few times too, and I've met a lot of other people in my life who've express that trait from time to time too, intentionally or otherwise.
I think there is an inherent trade off between 'confidence', 'expression', and of course 'a-priori bias in the input'. You can learn to be circumspect when you are unsure, and you can learn to better measure your level of expertise on a subject.
But you can't escape that uncertainty entirely. On the other hand, I'm not very convinced about efforts to train LLMs on things like mathematical reasoning. These are situations where you really do have the tools to always produce an exact answer. The goal in these types of problems should focus not on holistically learning how to both identify and solve them, but exclusively on how to identify and define them, and then subsequently pass them off to exact tools suitable for computing the solution.
I'm not sure why but the assumptions and naivety in this opening line bothers me. There are plenty of goals and problems that orders of magnitude more people are working on today.
- organic data exhaustion - we need to step up synthetic data and its validation
- imbalanced datasets - catalog, assess and fill in missing data
- backtracking - make LLMs better at combinatorial or search problems
- deduction - we need to augment the training set for revealing implicit knowledge, in other words to study the text before learning it
- defragmentation - information comes in small chunks, sits in separate siloes, and context size is short, we need to use retrieval to bring it together for analysis
tl;dr We need quantity, diversity and depth in our training sets
- LLMs aren’t very good at large scale narrative construction. They get too distracted by low level details that they miss the high level details in long text. It feels like the same problem as stable diffusion giving people too many fingers.
- LLMs have 2 kinds of memory: current activations (context) and trained weights. This is like working memory and long term memory. How do we add short term memory? Like, if I read a function, I summarize it in my head and then remember the summary for as long as it’s relevant. (Maybe 20 minutes or something). How do we build a mechanism that can do this?
- How do we do gradient descent on the model architecture itself during training?
- Humans have lots more tricks to use when reading large, complex text - like re-reading relevant sections, making notes, thinking quietly, and so on. Can we introduce these thinking modalities into our systems? I bet they’d behave smarter if they could do this stuff.
- How do we combine multiple LLMs into a smarter overall system? Eg, does it make sense to build committees of “experts” (LLMs taking on different expert roles) to help in decision making? Can we get more intelligence out of chatgpt by using it in a different way in a larger system?
What I mean is that every part of the output of an LLM should be annotated with references to the content that is most important or relevant to it.
Who is leading this effort now?
I don't know if that is the direction, but just an example that comes to mind easily.
If someone figures out how to do this, I think their models will be far more capable and reliable.
To truly eliminate hallucinations, I would think you'd have to change the initial training phase. Rather than only feeding raw text and predicting next tokens, you'd need to feed propositions labeled with some probability that they are actually true. Doing this with real fidelity is clearly not possible. No one has a database of all fact claims quantified by probability of truth. But you could potentially use the same heuristics used by human learners and impart some encoding of hierarchy of evidence. Give high weight to claims made by professional scientific organizations, high but somewhat lesser to conclusions of large-scale meta-analyses in relatively mechanistic fields, give very low weight to comments on Reddit.
That is all entirely possible but the manual human labor required seems antithetical to the business goals of anyone actually doing this kind of research. Without it, though, you're seemingly limited to either playing whack-a-mole with fine tuning out specific classes of error when they're caught or relying on a dubious assumption that plausibly human-generated utterances you're trying to mimic are sufficiently more likely to be true than false.
This problem arguably goes away if people treat LLMs for what they are, generators of strings that look like plausible human-generated utterances, rather than generators of fact claims likely to be true. But if we really want strong AI, we clearly need the latter. There is a reason epistemologists have long defined knowledge as justified true belief, not just incidentally lucking into being correct.
I wonder if a deterministic but natural language API would be any better for LLMs to integrate with? Or do LLMs already speak Json well enough?