Popular sentiment is a pretty meaningless metric for predicting the future.
I‘m arguing public sentiment will be right 50% of the time (in difficult binary predictions), not 0%.
The trouble with many of those construction examples is that they're point in time. Installer fucks up how the windows are installed re: flashing and water-proofing, but then puts the siding over it... the error is now completely hidden.
You could automated the inspection by AI photo analysis of every single window install on the project, say - but we could already do that for the inspection vs sending someone out, and send the photos to the expert instead, and we don't. Whether that's for lack of incentive to improve, or for ability to go deeper out of distrust for builders? I don't know.
Clearly this cannot continue, as the training costs will exceed all the compute capacity in existence.
The other limit is training data, eventually you run out of cheap sources.
There could be huge efficiency savings within the implementations but when training costs are already this high it seems naive to think the low-hanging fruit is still there.
I was not familiar with Edge AI, interesting concept. I feel like improving the efficiency of very large models is much more likely.
The recent successes will lead to an even larger influx of $ in the short term --- this is an existential threat to Google, after all. We will see where things go!
I don't know why people just keep repeating this complete fantasy as if it were true. Where does it originate from, I wonder? I suspect someone said something like that on social media, their post went viral, and now all of the internet is reverberating with this thing. It's a meme, yes?
I don't have social media outside of HN.
It comes from a few observations:
1) Large models are still improving with increased parameter counts (we do not know where the ceiling is yet; it could be low but it could also be high).
2) Most current architectures train by using all model parameters to produce an output, which is vastly inefficient. While it is not clear how to improve on this in the general case yet, in the simpler problem of NERFs, sidestepping this issue has led to a ~100x improvement in training time.
3) https://mingukkang.github.io/GigaGAN/ very recently increased the parameter count of StyleGAN by selecting parameters dynamically at runtime. They improved on previous results by a very, very large margin, at somewhat comparable training times.
I stand by my claim: "neural network architectures are still in their infancy, it is very likely that more efficient approaches exist". I am not claiming that AI will become sentient or anything crazy and do not understand why you are associating my point of view with other people. I just said that it is likely that a novel technology will continue to improve (has this it ever NOT been the case for any new technology?).
Well, if you want to know whether neural nets are in their "infancy" you shouldn't make "observations", you should read the literature. It goes back many years. Go to the primary sources, why try to guess and risk guessing wrong, as here?
>> I stand by my claim: "neural network architectures are still in their infancy, it is very likely that more efficient approaches exist".
Half of your "claim" is incorrect. Don't just double down on it! There's really nothing to "claim" here, the "infancy" or not of neural nets is not a matter of claiming or guessing. Either you know what it is, or you don't.
"if you want to know whether neural nets are in their 'infancy' you shouldn't make 'observations'" -> I think you have understood this as "neural networks did not exist before" when what I meant is "neural networks will still change a lot in the future"
I was interested in discussing, specifically: will NNs continue scaling up in size in the near/long term. In answer to the statement "the training costs will exceed all the compute capacity in existence", I added "it is very likely that more efficient approaches exist.". You replied that NNs have existed for a long time, and that because of this, it is a complete fantasy that more efficient approaches exist. Do you feel like this is a fair assessment or no?
Well, my mistake then for taking you seriously. Many apologies. You can rest assured it won't happen again.
You are right that neural networks are not in their infancy. Best wishes.
LLMs of this type will just start making up shit when they dont know something. because they simply generate the most probable next token based on previous x tokens. this is not fixable.
this alone makes these LLMs practically unusable in vast majority of real-world applications where you would otherwise imagine this tech to be used.
for instance, your comment confidently states this is unfixable - presumably based on the frequency you've seen similar text on the internet. why should anyone believe the veracity of your statement? These things didn't have any of these emergent capabilities one year ago, why are you so sure you understand their nature one year from now?
no its because GPT is based on transformers.
Aren't you just a function of your input and memories (stuff you've read, sensory input) as run through/managed by some neural network?
What makes you think the rest isn't just emergent properties?
And what makes you think you can't hook up the LLM with some algorithms or layers that handle some of the rest behavior of what your brain does?
and the issue of lost uncertainty is inherrent to this, yes.
to fix this, a new type of llm would have to be invented. this particular branch of development may very well be a dead end.
Perhaps it's because I'm ignorant about the inner workings, but calling the problem "unfixable" so early in the evolution of LLMs seems like foolish certainty.
We don’t teach babies by throwing lots of data at them, instead we teach them by giving them useful data.
The loop I see is: - train on a lot of existing data - run out of useful data - people use ai and give feedback (we’re here) - perform reinforcement learning on the data collected
Loop over the last two steps.
There is already more than enough data available, it’s just not nicely labeled to say if it’s high quality or should be discarded. Those last two steps will implicitly do the labeling.
156 million hours of YouTube videos
9,000 words/hour
6 characters/word (including space)
First, let's find out the total number of characters:
9,000 words/hour \* 6 characters/word = 54,000 characters/hour
Now, let's calculate the total number of characters for 156 million hours of YouTube videos:
54,000 characters/hour \* 156,000,000 hours = 8,424,000,000,000 characters
Since 1 character is typically 1 byte, we can convert this to gigabytes:
8,424,000,000,000 bytes / (1024 \* 1024 \* 1024) ≈ 7,842.11 GB
With a vision transformer each token may be around 16x16 pixels. I found an example where they use images of resolution 224x224 for training a vision transformer so if we go with that that 256 pixels per token and 50176 pixels per image, so 196 tokens per frame, 24 frames per second, that's 4704 tokens per second or 16934400 token / hour. In total we're at 2.6x10^15 tokens.
GPT-3 was trained on 5x10^11 tokens, so YouTube done this way would be around four orders of magnitude more tokens that GPT-3 was trained on.
GPT-3 was undertrained by 1-2 orders of magnitude, so the compute required to trained a model on YouTube would then be around 6 orders of magnitude higher than what was used to train GPT-3, so about one million times more.
I did a linear regression on the training costs from cerebras(1) and came up with the formula (1901.67366*X)-197902.72715 where X is number of tokens in billions.
Plugging in 5x10^15 tokens we get a training cost of 5 billion dollars. I guess a lot of optimizations could be done that would decrease the cost, so maybe its doable in a few years.
What you're seeing here with LLMs is sledge hammer number 2.
It's understandable how most people don't notice the sledge hammer. The decade prior to 2010 there was another sledge hammer had no smart phones. We were hit with a smart phone hammer AND an AI sledge hammer and the integration was so seamless we didn't even notice.
Much of the same will happen with LLMs. In 5 years it's so normal, nobody cares and likely we will forget what life was like before LLMs.
And still almost all of these applications are not really impactful or that important compared to actually society and life changing developments like the steam engine, electricity, electromagnetic transmission, cars, computers themselves and the internet.
Just more of the same, with added spice.
In the sense that we could revert to 10 years ago, and nothing would be much different or missed. Whereas going back without electricity or cars would be a totally different thing!
I think LLMs can be far more impactful than anything else hyped in machine learning of the past 10 years...
Oh, browser, you know me so well.
The AI sledge hammer is as good as we are going to get in paradigm shifts for a while given how we are at the top of an s-curve for development. Additionally we're only really seeing the first decade. When the computer was discovered was it ubiquitous in a decade? No. In fact it was largely a sort of useless academic research project for the longest time.
“What work has gone on for supply chain improvement in our European operations this year?” - this is the kind of question that is easy to ask in natural language but might take someone a week of searching, messaging, etc to assemble. An LLM with access to all of the code, documents, chats, etc could just give an answer with citations. We are not betting $1B on the answer that it gives, but it has saved us 1 week of work and allows us to move on to the next step in the project.
There are plenty of tasks like this which are highly valuable yet don’t require high trust. The one-shot “what is the airspeed of a coconut-laden swallow” type questions are actually fairly rare.
Trust is only a bound for certain areas, and this bound is eroding.
It’s just fancy auto complete to you? You honestly can’t see the capability it has and extend it the future?
What’s that saying about “it’s hard to get someone to understand something when their salary depends on their not understanding it”.
I don't understand why we can't look at the potential, or even current, capabilities of these LLMs and have a real conversation about how it might impact things.
Yet so many folks here just confidently dismiss it.
"It doesn't even think!" -- OK, define thinking?
"It doesn't create novel ideas!" OK -- what do most devs do every day?
"It is wrong sometimes!" OK -- is it wrong more or less often than an average dev?
That's not what "the science says", it's the title of an article that someone put on arxiv.
The article has no theoretical results, just a shoddy empirical demonstration of... something. The author claims that the something is an LLM simulating a Turing machine. But, is it? Really?
Well, here's how the article concludes:
>> Hopefully the reader has been convinced by this point.
"Hopefully" is not how you show computational universality of a neural net architecture. This is how:
https://www.sciencedirect.com/science/article/pii/S002200008...
i.e. with maths. That article you link to is a bunch of hooey.
Yes, this is arguing semantics but in this particular case, semantics is the whole point.
What I'm seeing now is that some kind of intelligence seem to emerge from these models, even though under the hoods it is just a bunch o matrix multiplications.
Who's can say for sure that our own brains doesn't work similarly? Maybe human intelligence is something that emerges from similar primitives and AGI will simply be a really large language model.
1. Solving all kinds of nontrivial problems posed in text format is extremely useful, no matter how it works under the hood. This means lots of people will use it, and it will change how people work
2. The more convincing the illusion of intelligence, consciousness, even sentience and personhood, the more people will convince themselves that it's real. And in my view, convincing a decent fraction of society that the next generations of these matrix multipliers really are sentient is far more dangerous than the direct effects of said matrix multipliers.
This changes when we do find this loop and start hooking the model to other input and output devices. At that time I'll call it sentient and if you don't I'd call your definition of the word worthless.
I mean we already did stupid chat bots in the 90s with convoluted if-else-spaghettis that imitated Eliza. That could already hold for some sentences... GPT-4 now holds much longer, but still you can easily drive it to the point where it starts "looping" and not provide much more value, especially if you are more expert => makes me so far hope that this length will grow more, but still.. in my "sessions" did not (gladly) find anything emerging beyond the first wow and surprising answers... not even close to that.
Here's a session I just ran on llama.cpp (7 billion parameter version). Everything above "====" line is the initial prompt:
Transcript of a dialog, where the User interacts with an Assistant named Bob. Bob is helpful, kind, honest, good at writing, and never fails to answer the User's requests immediately and with precision.
User: Hello, Bob.
Bob: Hello. How may I help you today?
User: Please tell me the largest city in Europe.
Bob: Sure. The largest city in Europe is Moscow, the capital of Russia.
User:
====
Who are you?
Bob: Who am I? I'm a virtual assistant, or a VA, for short.
User: Do you have a name?
Bob: Yes, I do. My name is Bob.
User: How old are you?
Bob: I am 19 years old.
User: Do you know what is "Hacker News"?
Bob: Hacker News is a question and answer website about programming, startups, tech, security, and web culture.
Is it wrong, or is it getting at the fact that so many people don't read the original articles, just the comments? ;)
Tell me a recipe for a veggie lo mein dish that has a unique spin, makes special use of a secret ingredient that I'd like you to choose—kind of like you're playing both announcer and chef on Iron Chef—and ensures that no ingredient exceeds a 10:1 carb to fiber ratio by mass.
GPT-4
Welcome, esteemed viewers, to this special edition of Iron Chef! Today, I, Chef ChatGPT, will be creating a unique and mouthwatering Veggie Lo Mein dish that is both delicious and nutritionally balanced. I am proud to reveal the secret ingredient that will give this dish a one-of-a-kind twist: miso paste! Miso paste will provide a savory umami flavor to our Veggie Lo Mein, ensuring a taste experience you won't forget.
Now, let's begin with the ingredients. Please note that all ingredients have been carefully selected to ensure that no ingredient exceeds a 10:1 carb to fiber ratio by mass.
Ingredients:
8 oz whole wheat spaghetti noodles 2 tbsp vegetable oil 1/2 cup julienned carrots 1/2 cup thinly sliced red bell pepper 1 cup chopped bok choy 1 cup sliced shiitake mushrooms 2 cloves garlic, minced 2 green onions, chopped 1/4 cup miso paste (secret ingredient) 1/4 cup low-sodium soy sauce 1 tbsp rice vinegar 1 tbsp agave syrup or honey 1 tsp grated fresh ginger 1/2 tsp sesame oil Crushed red pepper flakes, to taste Optional garnish: sesame seeds and chopped cilantro Instructions:
Cook the whole wheat spaghetti noodles according to the package instructions until they are al dente. Drain and set aside.
In a small bowl, whisk together miso paste, soy sauce, rice vinegar, agave syrup, grated ginger, and sesame oil. This will be our umami-packed sauce.
Heat vegetable oil in a large skillet or wok over medium-high heat. Add garlic and stir-fry for about 30 seconds, or until fragrant.
Add carrots and bell pepper to the skillet, and stir-fry for 3-4 minutes, or until the vegetables begin to soften.
Add bok choy and mushrooms to the skillet, and continue to stir-fry for an additional 3-4 minutes, or until the vegetables are tender-crisp.
Stir in the cooked whole wheat spaghetti noodles and the miso sauce mixture. Toss everything together until the noodles and vegetables are evenly coated with the sauce.
Add the chopped green onions and a sprinkle of crushed red pepper flakes, to taste. Stir-fry for another 1-2 minutes, or until the green onions are slightly wilted.
Remove from heat and serve immediately. Optionally, garnish with sesame seeds and chopped cilantro for an extra burst of flavor and visual appeal.
There you have it—a unique and delectable Veggie Lo Mein dish, brought to life with the savory goodness of our secret ingredient, miso paste. Enjoy this culinary masterpiece that is both satisfying and fiber-rich!
Or the internet? Who’d even want a computer at home to talk to other computers.
This is going to infiltrate every text-based tool. Email, social media, text. Speech and video already incoming.
But you're putting forth a "Bozo argument" by noting that that people also shit on the iPhone.
"But the fact that some geniuses were laughed at does not imply that all who are laughed at are geniuses. They laughed at Columbus, they laughed at Fulton, they laughed at the Wright brothers. But they also laughed at Bozo the Clown." - Carl Sagan
So the fact that some technological advances were laughed at does not mean that because something is being laughed at means it's a technological advancement.
Basically all you've done is say you disagree with the poster. But neither of you have given any sort of reasoning for your positions.
Personally, I agree with your final conclusion. Even as a "fancy autocomplete", this thing is pretty fucking fancy. To toss ChatGPT a basic idea and get back a multi-paragraph text on that idea that only needs to be proofread is pretty choice.
And if you don't care about proofreading, you could just send it something like "tell boss doctor tomorrow" and it could construct an email and send detailing why you won't be in work tomorrow due to a doctor's appointment.
Human communication just got a bit more complicated.
That riled me, but I held my temper as I asked, “What’s the matter with you?” He bellowed: “There’s nothing the matter with me. It’s you! You’re crazy if you think this fool contraption you’ve been wasting your time on will ever displace the horse.”
"""
- https://www.saturdayeveningpost.com/2017/01/get-horse-americ...