Google employees label AI chatbot Bard ‘worse than useless’
theverge.com
theverge.com
The one product I'm absolutely disappointed in is Adobe Firefly. I just got access a couple days ago and this is the most alpha product wrapped in pretty ribbons I've seen in a while. All the cool features they teased like text-to-vector and sketch-to-image/vector are not available. In fact, only 2 out of the listed 12 features work, and while OK, they are not better than other options. Also take into account there's no prompt history so good luck getting back to an image set you liked and the biggest head scratcher is you can't use it for commercial purposes. This is a design product. From Adobe. And you can't use it for commercial purposes... Who would ever pay for this?
I found it Firefly quite good for food, portraits...
We have a _very_ long way to go to AGI. Diluting down the meaning of the term "AGI" just to say we've reached it is, almost literally, moving the goalposts.
... not that I don't disagree with the rest of what you think you've said here, but I think that this part of your claim is egregiously paucious.
For every 20 paragraph thesis about the world around us, how often do you think someone meaningfully innovates and changes meaningfully our understanding of the world around us, and it's fundamentals?
Consider, like, classical logic. It's pretty much done. You have the universal gate sets, and any novel functions are just going to be an mxn-mux. Most other such general concepts are essentially figured out and have had their applications recorded, explained, demonstrated.
Sure, biographical information and history changes on a whim, but no one is requesting artificial clairvoyant intelligence.
I suppose you could have the model pontificate and assess the truth of its own argumentation (ask it to generate outer-product concepts, ask it to reassess assumptions in current models) but it will quickly learn its own language that will be mutually unintelligible with the corpus of current human knowledge it has been trained on.
Why does the underlying model need to change on a whim, in your opinion?... on what whim?
It’s not only textbook ‘confidently wrong’ (which is easy when you have strong opinions in a rapidly moving field) it also is a great counterpoint to ‘but LLMs hallucinate” because humans do that all the time too as seen here.
Spoiler:
- GPT4 can be given arbitrarily large long term memory via tools/plugins, vector stores, etc. It matters not whether that’s a feature of the model or an external tool made available to it, the result is the same.
- It’s completely unbounded by what it was trained on because it can be given tools to access any data, from google to specialized databases. It matters not if it has that feature in the model or it was made available to it as an external tool
- It can absolutely prioritize and plan, you literally just have to ask it and loop on itself. You missed LangChain and ChatGPT plug-ins to only name a few prioritization frameworks.
- There is absolutely no evidence that it cannot be creative and plenty evidence to the contrary. To deny this one has to create a definition of creativity that excludes AI. With access to an almost infinite supply of cultural information, it can expose you to novel ideas reframed given the right set of tools and temperature.
Sure we can pretend there is a universally accepted definition for AGI and even pretend it’s the one you have chosen - but it’s like arguing with a Taifoon on whether it’s a Hurricane. In terms of outcomes - all the same.
GPT-4 can't perform simple classical planning tasks so it's doubtful that it can "plan" in a looser way, as I think you mean it:
An "AGI" that has to be retrained from scratch, with more data and more compute, and with a modified architecture, every time we want to improve its performance? An "AGI" that forgets every interaction as soon as it's over? Doesn't sound too general, or too practical, to me, and certainly not very much of an "intelligence".
If OpenAI or anyone else finds a way to get their models to train themselves over and over at minimal cost, like humans do, then we can maybe talk about a "general" "AI". Until then all they have is a system with a static performance that does not improve with experience, that is not even machine learning anymore and that is very far from whatever people imagine when they say "AGI".
Transformers/self-attention solve the problem of scaling capabilities to the available data/compute. The big problem left is to have the right architecture such that AGI is in the solution-space.
>> The big problem left is to have the right architecture such that AGI is in the solution-space.
They'd have to train with AGI examples for that, though.
In my view, we will know we reached AGI when a network can learn to play chess well through self-directed analysis after only being given the description of the game. Obviously chess is a solved computational problem, but only through direct programming of minimax search. If we can teach a network with enough generality such that it can self-induce the right computational structure within itself to learn, e.g. induce a self-play subnetwork, then this is plainly AGI, superintelligence even. But I see a direct path from where we are now to such a network. This is why I think AGI is imminent.
MuZero, one of the successors of AlphaGo, learned to play chess (and a few other games besides) better than its predecessors while only receiving feedback about the end-state of a game (win, loss, draw), and not even the rules of the game, so it basically learned to play just by initially playing many (many) illegal moves. No "description of the game" needed. That didn't lead to AGI.
Progress in game-playing AI has not led to any progress outside game-playing. It's easy to see why. For chess and other classic board games there are specialised algorithms (minimax and Monte-Carlo Tree Search) that are sufficient to dominate any opponent, given enough computing power. Those algorithms are useless outside of board games (even for card games they're no good and dominant players use different approaches). For computer games such as Atari, where AI players often have superhuman performance, the complexity of the game world is tiny, compared to the real world, and techniques (Reinforcement Learning, basically) do not transfer easily from one to the other.
AI pioneers, like Alan Turing, John McCarthy, Claude Shannon and Donald Michie were interested in game playing as a gateway into understanding of the human mind, because it was assumed that the only way to play chess, and other classic board games, was to play them like the human mind plays them. It turns out, that's not the case. You can play very strong chess with a brute force search in alpha-beta minimax, and there is no human whose mind can process alpha-beta minimax well enough to beat a computer. That still tells us nothing about how the human mind works, or how to replicate it. What we have learned from all those successes in game-playing AI is that it's no use as a way towards general AI.
>> We do have a near endless supply of human generated data.
Yes, but what kind? We train systems on very narrow tasks: object identification in images, text prediction, speech recognition, document classification, etc. Those are not examples of "human intelligence", by which we could train a system to simulate "human intelligence", as a distinct process. They are examples of very specialised tasks that humans can perform using our intelligence, but it's not clear that intelligence is even needed for many of those tasks. That's why, for example, we have damn good image classifiers, but they are not AGI.
The problem with statistical machine learning algorithms is that they have to be trained to do one thing, and one thing only. They're one-trick ponies. If you want them to do multiple things together, you have to train them with examples of all those things at once. See for example Google's Gato, which had to be trained on multiple modalities as one big batch of token sequences. Google claimed that Gato is a "generalist agent", but it, too, was only capable of performing a restricted set of tasks, the ones it was trained on, and no others. Humans, by contrast, don't need to be trained on all modalities at once: we can learn to read, and later learn to play ball, and later still to drive. And each of those things is a hundred different tasks engaging different "modalities" (our senses). No AI system can do that.
AGI is not that easy as it perhaps might appear.
The issue of how the human mind works is tangential, but its also not true that computer chess has no relation to how the human mind works. Neural net based evaluations result in moves that more closely match human expert choices[1], which suggests a similarity in mechanism compared to prior evaluation mechanisms.
>The problem with statistical machine learning algorithms is that they have to be trained to do one thing, and one thing only. They're one-trick ponies.
LLMs show the beginnings of breaking out of this limitation. People who argue LLMs are just another example in the history of narrow AI don't seem to have a strong response to the example of in-context learning. The straightforward understanding is that we're seeing the beginnings of true generalization. I have not seen a convincing argument that in-context learning is just another case of what we've seen before.
Sorry. I thought you described a system that did what MuZero did. It still seems to me that's what you're describing.
"In-context learning" is a big potato. It's just some jargon that OpenAI came up with to play up the capabilities of GPT-3, just another example of the time-honoured tradition in AI of inventing grandiloquent terms to describe something much simpler. See Drew McDermot's "Artificial Intelligence meets Natural Stupidity", a paper from 1976, that castigates this already very common habit of AI researchers: https://dl.acm.org/doi/10.1145/1045339.1045340
To clarify, "in-context learning" is what we used to call "sampling from a model" back in my time (in 2014, when I did my Master's and studied neural nets and language modelling, and also game-playing algos; not coincidentally, because I usually engage in conversation about things that I understand, so you won't catch me talking about quantum computers, say, or physics). Models got better and sampling from them gets better results, so OpenAI figured they could embiggen it and call it something more impressive than "sampling". But it's sampling, not learning.
>> People who argue LLMs are just another example in the history of narrow AI don't seem to have a strong response to the example of in-context learning.
I don't think it's very productive to try and fit opinions into "people who say" kinds of boxes. I don't treat my interlocutors like that. Perhaps I haven't yet heard all the arguments on any particular claim. I'll never know if I assume everyone is just repeating the same old stuff.
Anyway there's no point in arguing anything here. If you've used language models before, you know how sampling works and it's obvious it's what OpenAI call "in-context learning". They're not even pretending it's not sampling. They're just claiming it's different because they're sampling from a large Language model.
Edit:
>> Neural net based evaluations result in moves that more closely match human expert choices[1], which suggests a similarity in mechanism compared to prior evaluation mechanisms.
The paper you linked describes a system that was explicitly trained to predict human moves. That's probably why it can do better at simulating the way humans play chess than a system that wasn't trained to simulate the way humans play chess.
I can see your reading of it. My point wasn't totally clear. What I was getting at is a system that automatically induces internal computational graphs based on its competence with natural language and its facility with solving planning problems. It's essentially a more general MuZero that is driven by natural language and can solve general computational problems over its entire learned world model. For example, when asking what is the 256th fibonacci number, instead of either retrieving from memory or confabulating a correct looking number, it constructs the right sequence of computational steps from its representation of the fibonacci sequence and runs through it. The difference is largely one of having the components of the system be self-directed through internal planning mechanisms, i.e. a cybernetic governor that can plan out general computational strategies. Natural language is a general representational framework and so a planning and computing facility over natural language representations is in my view a general problem solving framework.
>To clarify, "in-context learning" is what we used to call "sampling from a model" back in my time [...] Models got better and sampling from them gets better results, so OpenAI figured they could embiggen it and call it something more impressive than "sampling". But it's sampling, not learning.
The way I interpret this (analogous to sampling from a probability distribution), there is certainly a relationship between the two. But the difference is that in-context learning is much more general and flexible than anything that has come before. When the range of behavior far surpasses what has come before, its important to recognize the paradigm shift. There's a danger of expertise in old paradigms causing resistance to seeing these qualitative shifts. As Max Planck said, science progresses one funeral at a time. I can't help but see a case of this resistance in those who are downplaying the qualitative shift that in-context learning represents.
>The paper you linked describes a system that was explicitly trained to predict human moves. That's probably why it can do better at simulating the way humans play chess than a system that wasn't trained to simulate the way humans play chess.
The evaluation in the paper wasn't just for the network tuned to match human performance, it was also tested against Stockfish and Leela. Leela more closely matched human move choices at all human skill levels. It was also widely commented at the time that AlphaZero played a more human style of chess. The point is that this says something about how humans play chess, i.e. heavier on positional evaluation and light on calculation.
I asked chatGPT to create a meal plan, which it did, and all the recipes looked great, except, clicking the links for the recipes showed that they were completely made up, and either the links didn’t exist or were generated links, which in case they did were recipes for something completely different.
Doing the same search at Google may have required a couple of clicks, but all the top listing would return a weekly meal plan.
Lol. Nice try Mr Pichai
I haven't used Google in more than a month now.
If you want links to recipes by other people, I think Bing Chat would be more useful for you, since it can search the internet. There will eventually be a browsing plugin for ChatGPT, but it’s not available yet.
Why are you grading a chatbot on its web search result abilities instead of its answer-providing abilities? It’s like saying that horses are better than cars because the cars hurt when you sit on top with a leg down on each side.
Remember Google still has all the world's books digitised. Maybe the copyright laws that prevent them using it normally will allow it's use in a LLM? Just like every other model uses copyrighted works.