It is being played like a winner-takes-it-all right now (it may or may not be such a market). So it is a game of being the one that is left standing, once the others fall off. In this kind of game, speeding more is done as a strategy to increase the chances of other competitors running out of cash or otherwise hitting a wall. Sustainability is the opposite of the goal being pursued... Whether one reaches "AGI" is not considered important either, as long as one can starve out most competitors.
And for the newcomers, the scale needs to be bigger than what the incumbents (Google and Microsoft) have as discretionary spending - which is at least a few billion per year. Because at that rate, those companies can sustain it forever and would be default winners. So I think yearly expenditure is going to be 20B year++
Good reporting would present the key parts of the article up top, but invite you to read the full story if interested.
The lead paragraph and the inverted pyramid are key concepts developed in the field of journalism.
Not less efficient. But the impact on resources usage is still higher.
Of course the impact in terms of positive effects is also higher. So the cost/benefit may also have gone up.
I think that before that will be the PCB-ification, where you can order online and be batched in with others on a standardized process, for a reasonable price. This is actually starting to become possible, for example via TinyTapeout you can get tiny chips taped out for a few hundred USD.
Yeah many have their favorite gripes with $TheOthers, and hoping that the AI will solve them. It won't - not because AI isn't good, but because the people involved will quickly adapt. Likely making new annoying behaviors, and probably many will adapt to find new things to be annoyed at also.
Forced I don't know... But of course the financial incentives are very strong, as in many categories the Chinese brands have remarkable and sometimes astonishing value-for-money.
But for a small business, the cost of these tools might be quite low relative to manpower anyway, so paying 2x might not be a big deal. We got 8 Prusa machines at our local hackerspace, and 10 at previous startup lab I was at.
It would be good if the difficulty of getting patents would go up by a factor of 10. To get less of them in volume, and less bullshit ones. Should also throw out a bunch of the existing bad ones.
The TinyML niche, doing ML on microcontroller grade hardware (usually for sensor data analysis), is close in terms of constraints. Models measuring in kilobytes of RAM/FLASH.
But usually missing the flair and showmanship of demoscene, and the just-because-we-can attitude. So I agree, need more demoscene style!
As the maintainer of an open source library in this space (emlearn), I would be interested in contributing to such an effort.
GDDR6 8Gb spot (DRAMExchange) is now around 2.6 USD, down from 3.5 USD in summer 2023, and 6 USD in summer 2022? Last year has been pretty flat though!
Automation can be useful and valuable (economically) even if not intelligent.
Heck from a big picture view of solving a problem (say to manufacture something), then a solution/process/workflow etc that requires less intelligence may be the preferable one - if such a solution can be found, that is. It can be expected to be cheaper, more robust, repeatable.
This was a very important realization for me. That many times when I felt small cravings, the main thing I needed was not something sweet but water.
Before I would drink milk, lemonade etc. But now I just drink water. Helped to regulate blood sugar better during the day.
GPU compute per dollar has been on a pretty steady curve of around 10x per decade. In ML for computer vision we were also able to make models around 10x as efficient per 10 years.
I think with these two factors combined, mapped to LLMs, we will be able go match the performance of say Sonnet 4 on a 2000 USD workstation well within 10 years from today.
"LLM" as well, because coding agents are already more than just an LLM. There is very useful context management around it, and tool calling, and ability to run tests/programs, etc. Though they are LLM-based systems, they are not LLMs.
Seems like we may get local, open, workstation-grade models that are useful for coding in a few years. By workstation-grade I mean a computer around 2000 USD, and by useful for coding I mean around Sonnet 4 level.
Current cloud based models are fun and useful, but a tool that is / will be so core to the developer experience, I want to be able to run locally.
No, the open source code I make is there to be used freely and for all.
LLMs do not change this for me.
I think that open source libraries (but maybe not applications) may be even more relevant now than ever. More application code will be written, and there is still a need for correct and reliable components.
On the practical side there are many good options. Using the voice effectively is a key skill in many professions, such as singing, public speaking, education, religious, management. So there are many practitioners, and lots of learning materials out there. So I recommend to check of there are any of those angles that resonate more with you.
But very important - this is a challenge that needs practice to overcome - it cannot be dealt with only theoretically!
So as a simple starting point is: go on YouTube and search for vocal exercises. Start with something like warmup routines. And then try something like saying the same sentence at 7 different loudness levels, from a whisper into someones ear to a SHOUT that needs to go to another person at the other side of a lake. Observe just how much power you got!
This will be awkward and that is good - it means that you are stretching yourself, both physically and mentally! Like at the gym.
Then out a few sentences in the level you feel most comfortable with, plus 1 or 2 steps above. Try whole paragraphs.
PS: drink water and do not push harder if you feel pain.
The world models are not really useful yet. So they are starting lower, compared to LLM. So they probably have some decent gains to make still, before it gets really hard (diminishing returns).
Why is it like that, currently?
There is no information added by a vector index compared to the original text. And the text is highly redundant and compressible with even lossless functions. Furthermore a vector index is already lossy and approximate. So conceptually it is at least possible to have an index that would be a fraction of the size of what is indexed?
Hypothetically that might ne true. But current systems do not do online learning. Several recent models have cutoff points that are over 6 months ago.
It is unclear to which extent user data is trained on. And it is is not clear whether one can achieve meaningful improvements to correctness based on training on user data. User data might be inadvertently incorrect and it may also be adversarial, trying to out bad things in on purpose.
Logging everyones network data/metadata would likely be illegal under employment law in Norways. Other European countries may have same/similar rules. GDPR may also apply.
So be careful with how broadly you apply that default.
It has not learned anything. It just looks in its context window for your answer.
For a fresh conversation it will make the same mistake again. Most likely, there is some randomness and also some context is stashed and shared between conversations by most LLM based assistants.
I for one am quite surprised. Sometimes impressed. But also often frustrated. And occasionally disappointed. Sometimes worried about the negative follow-on effects. Working with current LLMs spans the whole gamut...
But for coding we are at the point where even the current level is quite useful. And as the tools/systems get better, the usefilness is going to increase quite a bit. Even if models improve slowly from this point on. It will impact the whole industry over the next years, and since software is eating the world, will impact many other industries as well.
Exponential? Perhaps in the same way as computers and Internet have been exponential - cost per X (say tokens) will probably go down exponentially the next years and decades, the same way cost per FLOP went down, on megabytes transferred. But those exponential gains did not results in exponential growth in productivity, or if so, the exponent is much much lower. And I suspect it will likely be the same for artificial intelligence.
A decent training pipeline will be able to reproduce models with equivalent aggregate performance (as measured by the evaluation metrics). And a high degree of similarity in behavior on specific inputs - but not identical.
It will not be the same exact weights. But that is not a critical bar to reach. And may software builds also fails that bar.
Probably a fair bit of hubris, sure. But right now it is not possible or legal to operate a company without a CEO, in Norway. And I suspect that is the case in basically all jurisdictions. And I do not see any reason why this would change in an increasingly automated world. The rule of law is ultimately based on personal responsibility (limited in case of corporations but nevertheless). And there are so many bad actors looking to defraud people and avoid responsibility, those still need protecting against in an AI world. Perhaps even more so...
You can claim that the AI is the CEO, and in a hypothetical future, it may handle most of the operations. But the government will consider a person to be the CEO. And the same is likely to apply to basic B2B like contracts - only a person can sign legal documents (perhaps by delegating to an AI, but ultimately it is a person under current legal frameworks).
Some of their core skill is taking credit and responsibility for the work others do. So they probably assume they can take do the same for an AI workforce. And they might be right. They also take do the same already for what the machines in the factory etc produces.
But more importantly, most already have enough money to not have to worry about employment.
No the preferred way of making modifications is the weights _together_ with training (or fine tuning) scripts, and the entire evaluation pipeline to measure performance. And the data required to support all of this.
When someone joins your data science team your would give them all this code and data. Not just the weights and say - the weights are the source, modify that to improve the model, I look forward to see your MR next week.
EDIT: Heck, sometimes the way to make improvements (modifications) is just to improve the data, and not touch the training code at all. It is often one of the most powerful ways. You still need training code though, and evaluation to measure the impact.
If everyone else had stayed silent as well, then I would agree. But as it is right now they are juuust about managing to match the current pace of the other contenders.
Which actually is fine, but they have previously set quite high expectations. So some will probably be disappointed at this.