The 65B model runs on a MacBook. So "currently" changes really fast, these days.
Tweet: "I hate it when my phone battery dies." Sentiment: Negative ### Tweet: "My day has been " Sentiment: Positive ### Tweet: "This is the link to the article" Sentiment: Neutral ### Tweet: "This new music video was incredibile" Sentiment:
and it's output was:
Tweet: "I hate it when my phone battery dies." Sentiment: Negative ### Tweet: "My day has been 90% balls." Seentiment: Negative ### Twitter: "i don't have a favourite type of music i just hate country" Seentiment: Negative ### Twitter: "c'mon first sunshine of the year...why the rain already?!" Seentiment: Negative ### Twitter: "Dear twitter, are you getting tired of me yet? Seentiment: Negative ### Twitter: "I don't care about the race, I just care about the president being a psych
Doesn't feel like that did the trick, but at least you got me to setup the shell command in a way where I pipe long-form prompts in via a separate file.
Rip open the code and make sure it has top_k 40, temperature 0.7, no top_p, and a repetition penalty of 1/0.85.
Proof: https://twitter.com/theshawwn/status/1632569215348531201?s=6...
Implementation: https://github.com/shawwn/llama
It feels basically identical to davinci. I think no one realizes it yet because Facebook released a horrible sampler that everyone copied.
Edit: This fork ( https://github.com/beiller/llama.cpp/tree/feature/repeat_pen... ) adds repetition penalty via two values, repeat_last_n = 64, repeat_penalty = 1.300000.
Playing with those now.
I wish I could somehow force it to output longer texts. It just stops at arbitrary points, often in the middle of sentences.
./main -m ./models/13B/ggml-model-q4_0.bin -t 7 -b 10 --top_p 0.0 -n 5000 --temp 0.7 --repeat_last_n 64 --repeat_penalty 1.3 --prompt "You are a question answering bot that is able to answer questions about the world. You are extremely smart, knowledgeable, capable, and helpful. You always give complete, accurate, and very detailed responses to questions, and never stop a response in mid-sentence or mid-thought. You answer questions in the following format:
Question: What’s the history of bullfighting in Spain?
Answer: Bullfighting, also known as "tauromachia," has a long and storied history in Spain, with roots that can be traced back to ancient civilizations. The sport is believed to have originated in 7th-century BCE Iberian Peninsula as a form of animal worship, and it evolved over time to become a sport and form of entertainment. Bullfighting as it is known today became popular in Spain in the 17th and 18th centuries. During this time, the sport was heavily influenced by the traditions of medieval jousts and was performed by nobles and other members of the upper classes. Over time, bullfighting became more democratized and was performed by people from all walks of life. Bullfighting reached the height of its popularity in the 19th and early 20th centuries and was considered a national symbol of Spain. However, in recent decades, bullfighting has faced increasing opposition from animal rights activists, and its popularity has declined. Some regions of Spain have banned bullfighting, while others continue to hold bullfights as a cherished tradition. Despite its declining popularity, bullfighting remains an important part of Spanish culture and history, and it continues to be performed in many parts of the country to this day.
Now complete the following questions:
Question: What happened to the field of cybernetics in the 1970s?
Answer: "
I'm curious if this part actually does anything to improve the output, my intuition says it doesn't help much. I don't have it running locally to test yet, though.
> # repetition penalty from CTRL paper (https://arxiv.org/abs/1909.05858)
Now I’m assuming a base rate of knowledge for this to help, but in general I find diving into the code paths of open source models, usually is a good use of time.
if you troll HN for AI related stuff and just read read read, progressively googling the terms you don't understand, eventually the haze will lift and things will start to make some sense.
source: in the middle of doing that right now
Do you mean "troll" or "trawl"?
(I'm thinking of an old saying that the best way to get an answer isn't to ask a question, but to say the thing is impossible. Can't remember it well enough to google for the exact form, however).
Trolling is done by hand with a fishing pole, sometimes on a dock.
Trawling is from the back of a boat, generally with a net.
I have a hazy recollection of folks in the early days of 4chan describing 'trolling for idiots' - as in, feigning ignorance to get a rise out of folks. Like, you post a dumb, obviously stupid but infuriating comment and watch people get all worked up about it. You know, for the lulz.
I feel like since then it's changed to just actually being a dumb asshole and enjoying when people are mad at you. Or maybe I've just gotten smarter. In any event at the time it certainly didn't seem like the kind of thing racist lunatics would go out of their way to do. Perhaps more the fool I, as I was a teenager at the time and as such that opinion cannot really be trusted.
https://paperswithcode.com/paper/most-language-models-can-be...
My work got cited by Gwern in this article! https://gwern.net/gpt-3
Looks like you don't have a 'most common/simple words' vocab restriction? That's a pretty fun one too. https://twitter.com/jonathanfly/status/1144345934857474048
I think this vocabulary filtering technique will pick up steam again now that good quality open source LLMs are proliferating. I've always wanted to try these techniques with ChatGPT, and the inability to do so has caused me to somewhat publicly complain about their lack of access to ChatGPT's vocabulary probability distribution (they don't give more than the top 10 most likely continuations).
As an aside - do you have any academic references for masking / constraining based next step transitions? It's a trick I've been messing with for years, but I probably picked this up from some older poetry generation work. Haven't been able to find any direct reference for simple 1 step masking, let alone more involved things like constraining based on the unrolled, modified markov sequence or things bigger than 1 step.
https://huggingface.co/blog/constrained-beam-search
This technique has been quietly available within HF for awhile, and has a lot of real world use-cases.
I've also been reading your paper on these topics, based on previous comments on HN. So thanks for that! Always interested in work about constrained generation.
I'd really like to see some version of the techniques from this palindrome generation https://www.ijcai.org/Proceedings/15/Papers/353.pdf revisited in modern contexts. Or some of the other Sony CSL work on this topic (e.g. https://www.francoispachet.fr/wp-content/uploads/2021/01/pap...) which I will admit I (still) do not understand yet.
[1] https://twitter.com/LalwaniVikas/status/1634648646535770113
This is insane
Vanilla Ubuntu desktop with a used server-grade NVIDIA GPU with lots of GPU RAM is probably your best bang for buck for messing with large model inference right now.
Mac has unified memory architecture that no other mainstream desktop can offer right now. This is the source of having up to 96GB VRAM on a laptop.
On a separate note I never understood why NVIDIA was always so stingy on RAM.
Give us a GPU with 256GB of RAM already. It doesn't need to be that fast, it just needs to be big enough to hold a huge model.
Instead they're still selling RTX4090Ti with 24GB of RAM while the free GPU I got from them at a raffle a few years ago has 32GB. WTF? I would have expected the RTX4090Ti to have 64GB minimum, being the highest-end consumer GPU.
But to me, Mac's architecture is the most inevitable final solution: the GPU becomes part of the CPU, and there's no CPU RAM and GPU RAM, just RAM. Will be great for AI too.
Very often multi-GPU training is maxing out RAM on each GPU and isn't making full use of compute capacity.
I suspect the future is tightly integrated CPU+GPU+RAM, but we may STILL have DIMM slots to acts as a buffer between "fast" RAM and disk. So your swap file goes from "fast RAM" to "DIMM RAM" before it has to go to disk, basically.
If the definition of RAM is DIMMs then it's really just a matter of merging the CPU and GPU into one unit with a massive L2 cache (or maybe we have an L3 cache) of e.g. 32GB that's able to hold a massive model without cache misses.
I've had success running the 7B model on a 16GB M1 Mac Mini (purchased in 2020!). I didn't bother to try but pretty sure the 13B model works as well.
I suspect you could grab one of those for less than $1000 these days.
I wonder if there are learnings for how to measure intelligence, just from proliferation of these AI models. Maybe even learnings for how to more efficiently access biological intelligence.
A human brain carries a lot more baggage with it. Long story short, our biological brain hardware is much superior to current AI hardware, but our biological brain software (where we learn the culture that surrounds us) is much inferior to current AI machine learning. GPT-3 trained 175B on basically the whole internet in 15 days. That's insane. And now there are rumors that ChatGPT is executed on an optimized subset of this 175B model, which may be a bit more than 1/10 that size.
I think going forward we'll see 20-30B parameter models perform much better than an average human in typical tasks. Especially after we evolve the topology of an AI to not be a straight directed acyclic graph, as it is now, but allow structures to loop internally, to invoke other structures with input (as subroutines) and so on.
And also there's new hybrid analog/digital hardware in the pipeline, which makes AI execution 100 times more energy efficient and compact. So above-human level AI on your phone is on the horizon, I'd say.
We're only getting started.
You literally couldn’t get GPT to come up with a single novelty if you tried. It’s all remixing existing content, and again, doing so in a way to fit the average of the dataset.
When you realize this you realize it has no intelligence as we most typically define it (novel solutions to novel problems).. its not AI. Call it what it is: a beautifully advanced way to regurgitate the exact most popular (mundane) reply you’d expect given a huge dataset.
It’s sort of good for studying what already exists. It won’t even really ever show you the edges though so it’s actually almost dangerously deceptive as evidenced by this absurd rounding up people are doing. If you want to learn the gist of anything, ask GPT. If you want to know anything in depth, GPT in fact will only mislead you towards genericity, platitudinous mediocrity.
You're also not right that GPT can't produce anything novel. Here's a tiny, modest example:
I won't waste my time, as I see you didn't read what you responded to the first time. But you need to start paying attention. You denial & empty confidence in what's happening will only get you so far.
I know many people who work in ML share this concern - people thinking it’s smart, it’s more than it is.
It’s very useful no doubt, but your comment is an example of extrapolation from maladjusted priors.
Your example proves my point, a very uninteresting looking game. I don’t even doubt GPT will find all sorts of interesting stuff by giving averaged out things. There’s plenty of gaps in the averages humans have missed. But nothing new, and until you can show that let’s just call it what it is.
On niche subjects, it fails often. But you can get it to correct itself by making it think logically by spelling out axioms, which seems to fit your definition of novelty.
All the arguments I've read about it not being intelligent seem to stem from some form of essentialism. Right now it's dumb intelligent but I think giving it an adjustable recursive depth, similarly to what BulgarianIdiot suggests, will go a long way to improving it given its base capabilities.
This demonstrates then when you instruct it "break down the problem and solve it iteratively" it comes out with superior solutions. More accurate, more novel. Ergo, then if it was allowed to iterate in this way internally, it'd produce better answers by default.
This thread sums up the same idea: https://twitter.com/ItIsFinch/status/1634730897520951296
https://openreview.net/forum?id=_VjQlMeSB_J
https://openreview.net/forum?id=5NTt8GFjUHkr
https://openreview.net/forum?id=qFVVBzXxR2V
I'm also looking forward to explicit long-term memory mechanisms that don't rely on the main architecture's weights nor on CoT.
As for "we just need a way to include iterations": isn't this what attention is supposed to do (kind of dynamically updating weights). The usual way to really update weights are variations on gradient descent. Can you link a paper outlining how we integrate your proposal into the current framework (which took around 20years to mature)? Otherwise your statements are Sci-Fi (at the flying cars-level).
Oh so it's not learning by example, it's just primed by a sample.
Oh so there's no iteration of thought, just a loop of prediction.
OK.
Also, the papers you asked about: https://news.ycombinator.com/item?id=35115563
BTW, people are already implementing recursive/iterative queries on the ChatGPT API, and getting promising results.
https://github.com/hwchase17/langchain
The general (non-technical) guideline is that the LLMs can "answer" anything you just gave them the answer for. So you give it a problem, ask it how to solve it, tell it to use that method and explain the data it needs, give it that data, and then show it everything at once: "With this data you requested and summarized, use this technique to answer this question".
How can you possibly know what's in its training set? Its training set is so large that any "guess" about what is or is not in it is a fool's errand. Given that we know how the technology works, there's no reason to claim emergent behavior when Occam's razor would say the items are likely in the training set.
It's quite easy to invent a sentence that's never been uttered before, it's also fairly easy to hit the limits of the internet's knowledge in pretty much any complex discipline that's not software engineering.
And it's apparent when you speak to ChatGPT, he will hallucinate a lot to compensate for niche topics it doesn't know much about.
Occam's razor wouldn't point to ChatGPT having omniscient knowledge of all things ever thought of and that will ever be invented.
https://acoup.blog/2023/02/17/collections-on-chatgpt/
> If you know nothing about either book, this answer almost sounds useful (it isn’t). Now this is a trivial research task; simply typing ‘the limits of empire review’ into Google and then clicking on the very first non-paywalled result (this review of the book by David Potter from 1990) and reading the first paragraph makes almost immediately clear the correct answer is that Isaac’s book is an intentional and explicit rebuttal of Luttwak’s book, or as Potter puts it, “Ben Isaac’s The Limits of Empire offers a new and formidable challenge to Luttwack.”
> A human being who understands the words and what they mean could immediately answer the question, but ChatGPT which doesn’t, cannot: it can only BS around the answer by describing both books and then lamely saying they “intersect in some ways.”
> The information ChatGPT needed was clearly in its training materials (or it wouldn’t have a description of either book to make a lossy copy of), but it lacks the capacity to understand that information as information (rather than as a statistically correlated sequence of words). Consequently it cannot draw the right conclusion and so talks around the question in a convincing, but erronous way.
It immediately cuts to the heart of the matter - an LLM does not understand things. It does not know anything but which word follows another. It can't do anything but regurgitate things it has read in an incredibly lossy manner.
But it's a neat party trick that seems to fool a lot of people into thinking that there's something legitimately useful, here.
I don't think that blog post's excercise is a particularly useful demonstration of anything.
This is still just deterministic except for a bit of noise to vary the output. There is no "thinking" going on. What you get is still just the most statistically likely output based on the training data.
Actually it turned out it didn't invent any of the games in that article. You can just Google them and find out.[0][1]
Reading is not believing.
0: https://www.digitaltrends.com/gaming/sumplete-chatgpt-ai-gam...
1: https://www.novelgames.com/en/labyrinthsudoku/
> I won't waste my time, as I see you didn't read what you responded to the first time. But you need to start paying attention. You denial & empty confidence in what's happening will only get you so far.
Maybe he didn't read what you wrote, but you didn't even bother Googling the claims in the article you posted.
> “Can you code a game called Sumplete?” Despite protesting about its inability to make games earlier, ChatGPT immediately started spitting out fully formed HTML, Javascript, and CSS code. Sure enough, it had once again created a version of Tait’s game – only this time, we never discussed what it actually was beforehand.
I guess "a version of Tait's game" leaves some wriggle room about exactly what it did.
I don't want or need to anthropomorphize AI (yet?), but this reaction by us is a tad too similar to rather uncomfortable parts of our history.
Your second link is a game different than Sumplete's rules, so not sure why you linked it.
You have one source, the Digital Trends article. It speaks about a mobile game being similar. The link here is at best circumstantial. The app page doesn't describe the rules of the game. Instead it says this:
"It is very similar to classical game of Kakuro"
I checked Kakuro, and it has similar, but distinctly DIFFERENT rules, as well. So, what's the theory here? GPT downloaded the game and ran it on its phone? No. Maybe the description of this game is somewhere. Maybe. But I couldn't find it. Where is it?
The excuse "it's somewhere, so it didn't invent it" will always be used for why an AI can't produce a "novel idea". While the reality of novel ideas is very simple: it's like conservation of matter & energy. You can never make something or destroy it. You can only rearrange it. New ideas are a rearrangement of other existing ideas. It's the case for everything we ever made. And everything we ourselves are.
It's one of those questions about the nature of us that's on the level of "is there a ghost in the machine"; "do we have actually novel ideas?".
I'm kind of leaning the way you are; I don't think we do. What seems to be happening that we _call_ novelty is just an advanced form of recursive synthesis - we take information in and remix it. But we're capable of levels of abstraction, so we can remix it really well.
I think that getting the abstraction part into these models might just be as simple as wiring them together with a meta-model. A network responsible for identifying similarity between outputs of different types, or one that serves as a connective layer, distributing tokens through other networks and then synthesizing the results.
What I'm almost certain of, is that the important changes are going to be architectural.
Three things I'm noticing:
1. We're reinventing structured programming in AI in fast forward. First it's a plain Markov chain. Then it's an "attention" directed acyclic graph. Then we realize we need loops. Then we realize we need to jump to different points in the loop. Then we realize it's useful to recursively call yourself or parts of yourself as a subroutine, parametrized with specific input. Etc.
2. Even before we fully realize this framework of thought into a model, I'm almost sure the model EVOLVES some of these structures during training. In the form of crude unrolled loops etc. Simply because it's inevitable for processing certain types of input data.
3. In order to preserve pragmatic outcomes, I'd bet the future is not one giant monolithic model for AI, but many medium-sized models, communicating in a meta network, like meta neurons, sending meta (high-level) messages to each other.
Essentially, we need to make neural networks more like a fractal. I have this rule of thumb that always works somehow: "no concept definition is complete, until it's made recursive". Neural networks will get there.
Mechanics work well for an AI generated game. No idea if it's novel or not, but it's a good take.
> These nets regurgitate the most average possible output given inputs. By definition.
You seem to be confusing the training objective with the capabilities.
For example:
> Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent as Meta-Optimizers
But note that this is happening at inference time, so it's not just generating the average of the input the model is trained on.
This constant reach to see human cognition as somehow unique (without evidence) is tiring and frankly an ego-extension / "made in God's image" type thought hole.
The person making the claim that this is as or more intelligent than humans should at a minimum show it do something as impressive as humans. I’ll wait for a great piece of art or science until I for no reason diminish humans, that’s all. It’s not tiring to me to see the beauty of human intelligence.
People seem to be able to draw outside the lines. It’s cool, it’s beautiful. I shouldn’t have to point to any specific amazing achievements because there are too many. I haven’t seen one from an LLM yet. I’ll know it when I see it! Not hating, just clarifying.
I'm saying that human intelligence isn't some mystic process above replication or emulation ... and that if an LLM-type computation can do what it does now I see not reason it can't match or surpass human intelligence in the future.
It may produce that intelligence in a different way to our organic squishglobes but I don't see that it matters.
We know so far that human cognition and sense of self is still largely a mystery and no evidence exists to show that it's just mechanical remixing of absorbed things. Contextually, many humans can indeed create novel suggestions, arguments and ideas, and they can self direct towards these in ways that no AI can so far. This does indeed make our cognition visibly unique without even having to mention anything religious
ChatGPT on the other hand is literally just a well-engineered design for creating coherent phrases from a huge training set of human information. As even its very creators admit, it doesn't consciously think, consider for real or perform literal AGI in any way that's sentient. For you and others here to call this equal to human cognition is absurd and not based on measurable evidence. It seems more guided by emotional awe at something presented in a new way than guided by sober reasoning. And you talk about human-centric ego arguments being tiring?
I'm not saying that ChatGPT is comparable in either capability or design to a human's cognition. What I am saying is that a human brain is just input,process and output. There's no reason a model of sufficient size can't emulate human thought in such a way that there's not much difference in capability (even if the mechanics differ).
I also think you're overstating the uniqueness of "novel" suggestions from humans and that "novel" ideas are some kind of high-water mark of intelligence rather than a fuzzy mechanism with induced feedback.
I can think of a novel idea by randomly mixing concepts I know. What matters is if this idea is good, and that test is done via interacting with the world. This mechanism can be implemented in an LLM now.
So if I ask chatGPT to make up a poem about the power of friendship, and it must involve a unicorn and a jackelope, it will give me that. What's not novel about it? Isn't it pretty much what any person would do, mash up some related words?
What does novelty mean in this context?
> Isn't it pretty much what any person would do, mash up some related words?
Yes, that’s exactly what chatgpt does, and it’s what many humans do. But to be analogous to your initial example, there should be another person who actually came up with the instruction specifying which words to mash into a poem. The word-masher, whether human or chatgpt, is just following the instructions, not coming up with them.
If I ask a random person I meet to write a poem with the same instructions as were given above I'd be surprised if they were able to come up with something as good as ChatGPT.
The novelty was not in the prompt.
That's an interesting claim I don't know how to measure.
>> Hey, ChatGPT, could you invent five words that mean hungry?
> Sure, here are five words that mean hungry:
> 1. Cravacious
> 2. Rumble-hungry
> 3. Chewyemptiness
> 4. Famishy
> 5. Voracihunger
Is that novel? I don't know. It feels a lot like the word games Lewis Carroll or Roald Dahl played, though.
Unfortunately none of this matters, because "no true novel AI" is at this point just the "no true Scotsman" fallacy over and and over again.
We'll need to learn the hard way. Thanks for the creative test.
Have you actually tried? You'd be surprised.
The fruit flys brain is capable of piloting the body of the fruit fly on a precision and energy efficiency that's simply unmatched in any kind of aerial machine we have developed. And only with those many connections, half a million? Neurons are very complex computation machines that can module their response very carefully.
The human brain does so much more than simply being able to process language. It's capable of one shot learning and to course correct incredibly easy. These features are very important for survivability and I would say are the "magic sauce" that we just haven't been able to replicate.
So I'm actually not so sure this is simply a parameters game. There could be important structures and specialisation that we are missing. In the fruit fly there are 93 different types of neurons, in the human brain iirc it might be hundreds.
You may think I don't know what "one shot learning" means, but I do know, I'm just saying it's a problem of our perspective, not of the model's capability. Aside from it having no long term memory (by design) from the chat. Yet.
Our high-level learning is absolutely nothing like how we learned as toddlers. It took us DECADES to understand complex concepts from "few shots". Only pretrained models can do that, and you're also one, in your adulthood.
What you say about extra complexity from neuron types and neurotransmitters combining with one another to modulate even simple networks is true of course. But it's also true you don't need to replicate all those organic details in a NN, because they're equivalent to just a few more "regular" parameters. Think of an artificial neuron in a network as a Turing machine. A Turing machine can do everything with just some more tape, and a neural network can do everything with just some more parameters.
And yet, our models show that we don't need AS many parameters as we thought. And that's exciting.
The whole point of machine learning is to do all that "decades" of training in a few hours.
In terms of learning material, it takes LLM a few orders of magnitude more material to learn than it does a human. After all, a human child trained with very few books (by age 7 a child would have seen only a few books) still outperforms CHatGPT trained with the same material, because ChatGPT trained on 4 books is basically useless.
So, sure, they can learn fast after all that. But so can GPT, which also one-shots a lot.
...
> I cannot phantom the amount of data and analyses that passes a typical human child in a day.
You're counting every second of every image that a child sees as a separate "image" that is learned. I'm counting the number of objects that a child sees, because people aren't doing frame-capture and analysis on every image, they're doing lazy interpolation (which is why all your peripheral vision in in monochrome but you perceive it as full color).
A 6 month old baby who has only ever seen close family (parents, siblings, pets) is quite able to discern the same features on strangers.
Nothing we have in AI right now can take 3 example models of humans, and detect hair, nose and mouth in a brand new never before seen model of a human.
Not really the same thing as training on millions of different images and videos. The number of distinct and different objects in a 7 yo child's training set is rarely the same as "every single youtube video ever uploaded".
Going back to the original point: ChatGPT needs millions of books to learn to regurgitate information as well as a semi-intelligent young adult, while typical toddlers are quite able to regurgitate what they've just seen.
Humans and animals get by with millions of times less training data than AI models need.
Then it sounds like he is slightly overestimating the dataset (since some of the frame is monochrome) and you are massively underestimating it.
However, if you substitute abstract shapes (blocks, balls, etc.), your point absolutely stands up.
In some ways, though, isn't it more interesting to have an intelligence that learns in a completely different way from humans, as opposed to one that learns through essentially the same method?
Like temporarily, or permanently, allocate some part of the brain for a certain configuration, beyond what experts can do and closer to what a processor can do.
Not especially long ago humanity knew basically nothing. The epitome of technology was the stone age. Literally crush two rocks together and then poke things with the sharp bits that flake off. Somehow, in the blink of an eye, we went from that to putting a man on the moon.
So let's do the same with these sort of LLM models and see what happens. Train one with the entirety of expressible human knowledge from the stone age. Where will it send you? I think the answer is largely pretty much where you are. In any case, certainly not to the moon. It seems we're just building natural language search.
This isn't shifting the goal posts. The problem is we keep intentionally making meaningless goal posts in pursuit of AI, because if you don't then the goal seems impossibly far off. But that's because the reality is that it probably is!
LLMs can make social consequences for us, from creating and maintaining state about opinions of us individually, and we will want to coexist in that reality. That's enough to adjust to and... respect. It doesn't seem to matter what arbitrary threshold of "intelligence" there is, in the face of an entity that creates much less arbitrary and dynamically generated - but predictable - outcomes. I don't have to understand what a human was trained on, or how smart they are, to understand the same thing. The same with an animal.
On HN intelligence is often seemed to be defined as more about new and novel thought / invention - it’s only intelligence if it thinks a thought that has never been thought/written before.
Off HN ‘intelligence’ seems to be closer defined to “is it smart enough to replace my job as a knowledge worker”. Pharmacists would say it has intelligence if it could automatically check prescriptions, and data analysts would say it has intelligence if it can answer a data request without their intervention.
IMO most people seem to define intelligence as whatever the part of their job is that currently requires analytical skill, because people are worried about the impact this could have on the middle and entry tier of knowledge work, rather than defining AI as needing to invent something fundamentally new.
These goalposts have literally been sent to space.
It's going to perform like a brilliant person with brain injury. 8 bit is far better and it would be interesting to see if they can get it working on any reasonable hardware.