OpenAI's new "Orion" model reportedly shows small gains over GPT-4
the-decoder.com
the-decoder.com
It seems clear to me that most value will come from little bits of LLM integrated into other applications. Code autocomplete is a good example of this, it’s valuable because it’s a key press away in the workflow we’re already in. Email editing clearly works best in an email client. Generating an image for a slide deck is most useful in the slide authoring application… and so on.
There’s plenty of product discovery to be done and value to be realised in this space.
Products are still going to increasingly capitalize on existing capabilities, like you say, but I wonder if the "AI everywhere" phenomenon might end up being a turn-off for a significant number of customers.
I wouldn't bet on a full-on ai wonter soon, but it starting to seem more and more likely the more hype companies are trying to build on it.
Though why was that fact never mentioned in the adverts? Or did i just not notice?
Writing Tools
Clean Up in Photos
Create a Memory film in Photos
Natural language search in Photos
Reduce Interruptions Focus
...
I mean its nice but stuff that mostly seems out there on the web already. The auto turning photos into film is mostly something I have to spend time figuring how to turn off. We shall see. Sticking my phone into 'focus mode' without telling me was one of the worst things Apple has ever done to me. I missed important calls because I expected my phone to work as a phone as usual.
Technically, only the US is supported. If you are British and you try to enable it, you’ll be told that it’s not available in your language. To get it to work, you need to switch your device language from English (UK) to English (US).
Apple says “English (Australia, Canada, Ireland, New Zealand, South Africa, UK) language support available this December.”
Since access is controlled by your language setting and not your location, I don’t think this is a regulatory thing. No idea why it takes Apple months to localise from American English to any other type of English.
I expect there will be, at some point, "trusted" or "verified" accounts that are able to flag a completion as wrong, and provide it with a correct completion, which will be collated and used (daily? hourly?) to fine-tune the weights. Active inference cluster would be switched as A/B/C etc.
I also happened to know a copywriter who said she was laid off cause of ChatGPT.
It still is going to be a disruptive technology, though it is not perfect by any means.
That's after immense resource consumption and cost from at least a year of training.
Is this when it happens? That we discover training with synthetic data only goes so far? We may not have the clear picture yet but the writing is appearing on the wall. In this case, the AI community may not have a clear path forward. This was supposed to be it.
For you investors sake, I hope you are watching. It really feels like we're moving into the next phase for the time being, a more mature one where we seek applications of AI rather than their evolution. I could already feel it earlier this year when more exciting stuff happened in small models for mobile devices.
Have they shifted the definition that much, or is there something they’re not telling? My impression is that, based on traditional definitions, no one has the foggiest notion of where to head to achieve it. And if he’s suggesting that “creative use of existing models” is a path to AGI… wow, that’s gotta be a useless definition of AGI.
Artificial.
Good enough.
Intelligence.
Besides, most people i know that use AI say they would get much more out of it with smaller more specifically trained models than the all knowing one we have now, but that depends.
I suspect stage 1 is to get current models to assist with making AI breakthroughs.
They probably have ideas and a research direction, but it's probably going too far to call whatever they have a plan to get to AGI, since no one knows how to get there. At best you could probably say they have a plan to search for a path to AGI.
> I suspect stage 1 is to get current models to assist with making AI breakthroughs.
That's sounds like a sci-fi trope.
OpenAI barely has any moat, I Seriously wonder why no one is talking about hoq much open weighted models are getting to the closes source ones, they need invesotr cash to build something that would let them stabd out, otherwise no one would bither paying for OpenAI API when you can just run your own model.
It can do this to a limited degree. I did mean "help with" in a very practical, limited way.
https://www.noemamag.com/artificial-general-intelligence-is-...
What most people mean when they say AGI is ASI.
How do you know what most people mean?
Update: I agree with the article that a reasonable amount of generality exists with chatGPT. I dispute the intelligence part of it. It cannot evaluate the truth or falsity of what it spouts out. This appears to be a fundamental limitation of LLMs.
> Have they shifted the definition that much, or is there something they’re not telling? My impression is that, based on traditional definitions, no one has the foggiest notion of where to head to achieve it. And if he’s suggesting that “creative use of existing models” is a path to AGI… wow, that’s gotta be a useless definition of AGI.
It's likely they're taking cues from Musk's "Full Self-Driving" fiasco: make big promises, then when progress stalls, redefine the promises to match what you've built.
Then again, we as engineers are always the annoying nerds who get stuck in the details.
A lot of traditional definitions imply some degree of consciousness or sentience.
But AGI will ultimately be defined by capabilities, not similarity to humans.
If a GPT-style LLM meets or exceeds humans at a broad category of tasks, it doesn't matter that it doesn't "understand" the tokens it operates on. Same with self-driving systems or autonomous robots. It doesn't matter if the CNN doesn't have a conception of a "person" - if it can feed the identification vector into the control network, and the control network swerves to avoid the identified object at a rate equal to humans, then that's good enough.
Capability is what matters, nothing else. And right now, we have ALL the parts to do this, we just have to scale them up, and train them better, and connect them.
If you'd showed one of these models to an AI researcher in the 90s or 2000s they would have been jumping up and down shouting "that's AGI!"
As we've been able to play with these models, we as humans can tell that there's something missing, a je ne sais quoi of human intelligence. They can think, they can reason, sometimes on very technical subjects, but the reasoning is shallow and fragile.
But to say that we are nowhere near AGI is definitely an exaggeration. We are on the precipice.
The next step would seem maybe to think about it, which o1 is maybe making a start on. If you think of humans learning say programing, they read the books and can answer questions but they have to do exercises, write code and so on which gives greater understanding which is maybe where AI needs to go next.
By the way the Altman Tan Nov 8th interview which is a source for most of the Altman mentions in the article is worth a view if you skip the boring bits https://youtu.be/xXCBz_8hM9w
GPT-3.5 with the ChatGPT front end was just such a massive improvement over everything that came before it, then 4 months later they release GPT-4 which was a significant improvement but more importantly gave the impression that progress was actually accelerating, then for the last 20 months we got incremental tweaks and new models that are a little better at certain tasks and a little worse at others.
So, chatgpt 3.5 potentially was to 3 as 4o and o1-preview are to 4, with 5 still to come, but it definitely could hit a data wall and maybe they lost some of the pirated book and scientific paper and textbook sources after scrubbing for the lawsuits, but they also have a lot more user interaction data and mass amounts of uploaded data from users that may not exist on the web.
They are now ranked in the top 10 websites in the world or close and are popular especially among knowledge workers leaking all kinds of data to them.
This story only matters if we find out scaling doesn't work. If some new experiment on context/cot/long-term memory etc. doesn't work, that tells us nothing
Considering he was willing and Delusional enough to ask for 7T USD for AI chips, Im sure he would try.
This is also ignoring the giant elephant that is open models, soon enough models like Llama, would be able to match or even surpass what ChatGPT, by which point why would any sufficently large company pay for API when they can run their own model, especially when all those GPU used for training flood the market.
But then again, plenty of large comapnies still use aws, even when it makes no sense to go serverless, so they might have a market to capitalise on.
We live in interesting times for tech, moores law is dead, intel is falling, layoffs are everywhere...
I sure picked the best time to go to university for Computer Science T-T
>I sure picked the best time to go to university for Computer Science T-T
Going to be a bit contrarian here – while it's true that jobs can be tough to find and layoffs are discouraging, I genuinely believe it's also one of the most exciting times to be in the CS field. The fact that we can actually talk with an "algorithm" is still quite bonkers to me cause I remember fiddling with RNNs and LSTMs just to predict the next word or two in a sentence. There are still ways that we can leverage it to make something really cool. Perplexity is one. Phind is another. Notions's AI integrations are great.
I graduated about four years ago and faced my own setbacks, including getting laid off from my first job. But despite those hurdles, I've managed to find my footing and am doing reasonably well now. Just hang in there champ.
True, I sorta conflated running llama on your pc with what large comapnies.
Not to mention how I was somewhat conflating chat-gpt the product with OpenAI the company, What i argues was that soon enough ChatGPT itself won't be that special when comparing it to open source models.
OpenAI the company is in the weird position of both having a moat, and yet drowning in it: They have a huge advantage in skilled experts, engineers, and know-how to get a first mover advantage, especially now that they are practically another subsidiary of microsoft.
But they also have the notable disadvantage of spending billions upon billions of dollars developing a model that in the end is little to no better than what one could get for free from the internet.
A small company with a few dozen specialist could present a comparable product at a fraction of the cost, simply by not having to pay back the cost of developing their own model.
I feel like OpenAI would end up in a weird place in soon, maybe something like a cloud provider for companies, usefull for smaller ones where brand recognition and reliability matter, but having to compete with more specialised companies offering a similar service using llama, And at some point large companies could just build their own servers with open-source LLM's with their own servers and their own teams, bypassing OpenAI entirely.
The biggest winner here is those new small AI consulting teams that didn't have to spend nearly as much on finetuning the models that are already made.
You probably know way more about these things than me, what do you think of this prediction?
It doesnt sound as terrible for developers as I first thought, though it pains me to see how many people quit/never went into software development due to the AI hype, we lost a third of our class from 2023, and I assune things are even worse in america/developed countries.
Despite these challenges, I believe OpenAI has strategic avenues to sustain and grow. Their investments in integrations, enterprise solutions, and reinforcing the reliability and scalability of their models can maintain their edge. The trust and infrastructure they offer might still be appealing enough for many businesses to stick with them, similar to the AWS analogy you mentioned.
As for the job market and the future for developers, I see your point. The AI hype has indeed introduced some volatility. However, I am cautiously optimistic. The evolution of AI and its integration into various fields will eventually balance out, creating new opportunities even as it displaces others. I still believe we’re in a transformative period where mobility and adaptation within the CS field could lead to exciting new prospects.
To your last point, it’s indeed tough to see talented individuals shy away from software development due to the current uncertainties. However, I hope this phase will pass and those who remain will likely find themselves at the forefront of some groundbreaking developments. Let's hope our lord and saviour, J-Pow has many more rate cuts for us in the future.
Still, feels like we are at risk of another AI winter soon, especially with such a huge bubble being built around it this time.
Here to hoping the US wont hit a recession bceause of it.
It is still exciting to see some breakthroughs in this field, but it isn't going to be like the early days of GPT-4 release.
Or is this a disconnect between pertaining loss and downstream performance on real-world tasks?
If there's no extra data most scaling predictions start to look bad.
None of this is surprising. Every AI/ML researcher has done a back of the envelope estimate to see how much data there is and when this must stop. Me included. It's a common topic at conferences.
What's your estimate for how many tokens that represents?
At some point the LLMs are just redigesting their own vomit.
3.5 Sonnet felt close but even that feels like it's being downgraded now.
Just feels like things are silently being quantized or downgraded in other ways to increase profitability once people jump on. Even trying same prompts that one shotted flawlessly pre-Dev Day 2023 now just fail for me on GPT.
I remember there was a forum post about it that got a lot of flack, but maybe they were onto something.
Here it is: https://community.openai.com/t/did-chatgpt-4o-get-progressiv...
Of all the gaslighting I see in Ml/AI, the idea that quantization is almost free is up there. I don’t buy it for one second at all, no matter how many charts you try to show me implying that the logprobs are 99% the same. Sure, maybe until you go to 10K tokens context window!
1) Creating new "harnesses" for models that connect to various systems, APIs, frameworks, etc. While this sounds "trivial", a lot of gains can come from this. Similar to how the voice version of ChatGPT was (apparently) amazing, all you really had to do was create an additional voice to text layer and another text to voice layer.
2) Increasing specialisation of models. I predict over time that end user AI companies (e.g those that just use models and not develop them), will use more and more specialised models. The current, almost monolithic, system where every service from text summary to homework help is plugged into the same model will slowly change.
We haven't seen wholesale specialised models yet because creating foundation models is expensive and difficult and the current highest ROI is to make a general model.
In what measure, loss? Loss can't go below 0 plus the inherent entropy in the text (other than that with overfitting it could reach nearer to 0, but not fully if it is next token and there are multiple same prefixes).
With respect to hallucinations 4 got incredibly better over 3
The inputs - data, compute and parameters - going into training these models have grown by many orders of magnitude between each gen. There's a lot of fuzziness about how much better each gen has gotten, but clearly 4 is not many orders of magnitude better than 3 by any reasonable definition. This mental model isn't useful to say how good each gen is, but it is quite useful to see the trend and make long term predictions.
What OpenAI were hoping for was AI stepping in to provide the breakthroughs. But for now it'll have to assist with incremental improvements only.
There's probably 50% disappointment from the AI optimists and 50% relief from the pessimists.
Also, intelligence isn't the only criteria. Updated knowledge also matters. Context length also matters for large problems that can't be divided.
I wonder what price these llm's can be run in order to be profitable,and whether just running your own model would be worth it.
Maybe it could be even cheaper if youre willing to fall behind on R&D, but keep in my mind that everytime openAI invented something, it was quickly copied by its competitors.
I agree that training LLMs will get cheaper, but it's likely that the compute bottleneck no longer limits LLM performance.
Unless OpenAI is willing to do something desperate, the best they have right now is what llm are, and so the cost would be in maintaing them. If you already paid for a bunch of H100's to train, there is little incentive to move away unless you know TPU are going to be significantly cheaper to run, cheap enough to explain the new cost of buying them.
This is ignoring the giant bubble that has balooned out of AI hype, which if popped would be disastorous for the comapnies most invested in the industry. Nvidia has a P/E ratio of 60-70, if they dont get enough future growth to explain it, they could lose a third of their pricing if not more.
There's also lots of utility to be found with the best LLMs today. I'm working on something myself, and have seen others pushing the boundaries in hackathons and startups. So that's a lot of innovation and value that's definitely not a bubble.
There's a lot of resources... my favorite is https://karpathy.ai/zero-to-hero.html
I wonder what kind of future llm have, since soon with a good enough GPU you can run your own model that is as good as the paid one(which i expect the price of to rise considerably once investment dries up)
One option is running ads i guess, though i question how that would even work.
Maybe thats why they are talking about a fully AI generated feed, despite how unpopular that is with users.
Dont have to pay tge content creators a share if you produce all the content, lol
If you get a big profitable following you even start having to pay them for boosts to keep it.