Advancements in machine learning for machine learning
blog.research.google
blog.research.google
These things are inferior at many levels: - Algorithmic: These things aren't feeding back to their human masters tips and tricks on how to modify the network to go faster beyond some very basic signals. - Loss of intent: ML network designers are specifying architecture in python, and by the time it's gone through many layers of lowering, you can get some complete garbage. Highly efficient garbage, but still garbage. (recent example, we caught one of these compilers doing a slice update operation by first forming the range of all possible indices to the array, slicing that to get indices to update, and then doing a scatter; we replaced it with a single memcpy call). - Inefficient kernels. Every time we see the output of these compilers go head-to-head with an expert assembly programmer, the compiler loses, often by 30%+. This always seems like the sort of thing that should be easy to solve, but given no-one seems to have cracked it in the past 50 years, it's obviously not as simple as it sounds.
Now consider all the heuristics for inlining, loop unrolling, vectorization etc in compilers, certainly a neural net can be beneficial and possibly easier to maintain than tons of human written heuristics.
I'd be very surprised if that can be extended to a large complex algorithmic system that is amenable to mathematical reformulations (at least within the next 10 years).
https://github.com/official-stockfish/Stockfish/commit/af110...
Funny you should say that. Because traditional compilers have been incredibly useful.
That throughput is the point though? You cannot have performance specialists on every single ML workload. It's still significantly better than not having these kinds of optimization.
What's the actual state of these "ML compilers" currently, and what is rhe near term promise?
You simply write model = torch.compile(model)
"Across these 163 open-source models torch.compile works 93% of time, and the model runs 43% faster in training on an NVIDIA A100 GPU. At Float32 precision, it runs 21% faster on average and at AMP Precision it runs 51% faster on average."[1]
What google is trying to do, is to involve more people in the R&D of these kind of methods.
It actually sounds very useful and cool, I just completely did not get that from the article.
Disclaimer: I will be very handwavey, reality is complex.
This is achieved by compiling the graph into some intermediate representation. And then implementing the right backend. For projects here, look at stableHLO, IREE, openXLA.
You can argue that Jax's jit compiler is a form of such compiler, mapping the traced operations down to XLA, which then does its own bit of magic to make it work on your backend.
It's transformations and abstractions all the way down.
This is only about predicting performance of a given graph and not about improving/suggesting/editing a new equivalent graph. As in FunSearch, models which have decent predictive power could be used with evolutionary search.
[1] https://github.com/google-research-datasets/tpu_graphs#featu...
[2] TpuGraphs: A Performance Prediction Dataset on Large Tensor Computational Graphs https://arxiv.org/abs/2308.13490
[3] Learning Large Graph Property Prediction via Graph Segment Training https://arxiv.org/abs/2305.12322
*1. Input:*
* Tensor shape: [2,4,16] * `2`: This represents the *batch size*, meaning there are two independent data samples being processed. * `4`: This is the *input feature dimension*, indicating each sample has 4 features. * `16`: This is the *input channel dimension*, suggesting each feature has 16 channels of information.
*2. Kernel:*
* Shape: [4,16,8] * `4`: This is the *kernel size*, meaning the filter window used to convolve has a width of 4. * `16`: This matches the *input channel dimension*, ensuring the filter operates on the same number of channels as the input. * `8`: This is the *output channel dimension*, indicating the convolution produces 8 new channels of information per sample.
*3. Output:*
* Shape: [2,8] * `2`: This remains the *batch size* as the operation is applied to each sample independently. * `8`: This matches the *output channel dimension* of the kernel, signifying the final tensor has 8 new features extracted from the input.
*4. How is it possible?*
Despite the seemingly mismatched dimensions in the input and output, convolution on graphs works by leveraging the *neighborhood structure* of the graph. Here's a simplified explanation:
* The kernel slides across the graph, applying its weights to the features of the current node and its neighbors within a specific radius. * This weighted sum is then aggregated to form a new feature for the current node in each output channel. * As the kernel moves across the graph, it extracts information from the local neighborhood of each node, creating new features that capture relationships and patterns within the graph.
*Additional considerations:*
* The graph structure and edge weights likely play a role in how information propagates during the convolution process. * Specific details of the convolution implementation, including padding and stride, might also influence the output shape.
Off the top of my head, I can think for at least five foundation models (Llama, Claude, Gemini, Falcon, Mistral) that are all trading blows, but GPT is still a head above them and has been for a year now. Transformer LLMs are simple enough that, demonstrably, anyone with a million bucks of GPU time can make one, but they can't quite catch up with OpenAI. What's their special sauce?
I’m speculating here but I think Google always refrains from getting into the manual side of things. With LLMs, it became obvious so fast that data is what matters. Seeing Microsoft’s phi-2 play, I’m convinced more about this.
DeepMind understood the properties, came up with Chinchilla but DeepMind couldn’t integrate well with Google, in terms of understanding what kind of data Google should supply to increase model quality.
OpenAI put annotation/cleaning work almost right from the start. Not too familiar with this but human labor was heavily utilized to increase training data quality after ChatGPT started.
Or how do you think Google evaluates search-ranking changes (or gather data for training various ad-ranking & search-ranking models).
Overview: https://blog.google/products/search/overview-our-rater-guide...
Full PDF: https://static.googleusercontent.com/media/guidelines.raterh...
If I recall right, GPT4 got done in October. After that, it was RLHF and safety work (Bing starts using GPT4 publicly in February, a month earlier than official launch)
Ultimately though, it's futile to argue which model got done first, as long as the models were behind closed doors. But ChatGPT launched before Bard did and that's the pertinent part that gave OpenAI the first-mover advantage.
No. A lot of people think it really matters
A lot of other people pretend to care about it because it also enables stifling the competition and attempting regulatory capture. But it's not all of them.
What makes you think it is for show?
It's outputs non-sensical (aka highly hallucinating) or relatively useless but coherent text.
It really needs further refinement.
This is one big reason why GPT-4 is still the most popular.
We cannot know truly how these parameters interact at large scale and also how they interact with each other.
Is it really the case that openai has data that Google doesn't?
It will be interesting to see how capable Gemini Ultra actually is. For now we wait.
https://deepmind.google/discover/blog/funsearch-making-new-d...
I'm pretty naive in terms of granular understanding here as I am barely proficient in Python, to be clear, but when I daydream about things you could solve with machine learning/AI, this is the approach I always think of and I guess is how I thought it already worked. Load it up with the best information we have currently, define the desired results as clearly as possible, implement some form of automatic feedback, and let it run iteratively until it produces something better than what you had before.
Is this a case of "well no shit, but actually implementing that effectively is the hard part"? Is it being able to quickly apply it to a wide variety of problems? I guess I'm trying to understand whether this is a novel idea (and if so, what parts are novel), or if the idea has been around and it's a novel implementation.
Otherwise the description is the same as "select randomly, keep the best, iterate".
The goal is also complex. You might be thinking of "find the most efficient program" but that's not what we're doing here iiuc. We're trying to get a program that makes other unseen programs more efficient. That's hard to define as a goal.
That is what they did though. The LLM didn't know what problem it was "solving".
They also don't just keep the best and search from that point but feed the resulting programs and their scores into an LLM.
The hype may be specific to some companies for now, but AI is deeply going to change many industries, especially due to open-source, specialized chips to allow running in local, and new hardware (I strongly hope a clone of H100 A80G comes quickly).
The next step is to add limbs to the LLMs.
Then we get Tesla bot who is going to help you with daily chores, and to execute tasks in a factory.
The bot can ask its own internal knowledge base to know what action to execute next, and because the model can output JSON, the action commands can be sent to motors for in-real-life execution.
You mean it isn't overhyped, hype is just what expectations people have it is underhyped or overhyped that says how those expectations related to reality.
I am hoping we get to the point where the models are good enough that classes in schools are introduced on how to use them rather than just build them as the number of people wanting to or willing to learn programming is a lot smaller than the number of people to looking for ways to do things more efficiently.
It reminds me of the middle ages where only the priest was allowed to read and interpret the bible, mostly through the virtue of knowing latin. Then suddenly the printing press comes around and everyone can get their own cheap bible in their language. You just can't fight and enforce this kind of thing in the face of such insane progress. In 100 years (if we're not extinct then) people will probably look back on mass education where one overworked teacher tries to explain something in a standard way to 30 people (over half of who are bored or can't keep up) as some kind of old age savagery.
I am certain in 30 years everything will still be the same.
I would agree, not a fundamental or radical improvement yet.
Will it be? I hope so.
GPT 4 is fine for absolutely foreign tasks to me, like write a power shell script, because I know almost nothing about power shell. However those tasks are rare and I generally competent about things I need to do.
I use it for TypeScript/React. But it’s just a more comprehensive code complete. Incremental.
If coding hasn't change much in 30 years, it may mean you have not changed much in 30 years.
I right fundamental software. Chromium, Node. It is good old, largely incremental, C++.
I like programming how I do now. I don’t plan to stop.
People do lots of things manually that machines have been able to do for a long time.
I don't think anything significantly changed in my approach to the code since 2013. We will see how 2033 goes, but I don't expect nothing big either. ChatGPT is just Google replacement, Copilot is just smart autocomplete. I can use Google instead of ChatGPT and I could use Google 10 years ago. I can use vim macros instead of Copilot. This AI stuff helps me to save few hours a months, I guess, so it worth few bucks of subscription, but nothing groundbreaking so far.
Right now we're still OK because the AI isn't good enough; when it gets good enough, doing things manually is as economically sensible as making your own iron by gathering a few times your mass in wood, burning some of in a sealed clay dome to turn it into charcoal, digging up some more clay to make a porous pot and a kiln to fire it in, filling it with iron rich bacterial scum from a creek, letting the water drain, building a furnace, preheating it with the rest of the wood, then smelting the bacterial ore with the charcoal, to yield about 7 grams or iron.
> Will we be 10,000 times more productive or will we be out of work? I think a lot of people will be out of work.
It can be both. Automation of farming means we've gone from a constant risk of starvation to an epidemic of obesity, while simultaneously reducing the percentage of the workforce in agriculture.
no one is going to be using AI and then just have it 'replace them', they're going to use it to augment their abilities and avoid replacement.
A manager can prompt a chatbot to write a thing instead of prompting me to write the same thing — for the moment, what (I hope) keeps me employable is that the the chatbot is "only" at the level of getting good grades rather than n-years professional experience.
I have no expectation for any specific timeline for that to change. Perhaps there are enough incentives it will never get trained to that level, but also perhaps it was already trained 4 months back and the improvement to capabilities are what caused the OpenAI drama.
You could easily flip it around, ask the bot to manage you better than your manager and make the best use of your time, or something like this?
But others your point is valid.
All disciplines evolve over time and those who fail or refuse to keep up will be left behind.
Sometimes quality and responsibility matter. Even if a machine is really good at producing bug-free code, often someone is going to have to read/understand the code that the machine produces and take resposibility for it, since machines cannot take responsibility for things.
In the end, you'll always need a human that understands the formal algorithmic language (i.e., a programmer), capable of parsing the formal program that will be run, if you want to be able to trust the program, since any way of mapping an informal request (described imperfectly in an informal language) to a formal construct is always going to be "lossy" and prone to errors (and you'll know this if you ever used an automatic code generator). Just because someone is willing to blindly trust automatically-generated code, doesn't mean everyone else is: there are contexts in which you need a person to blame, when things go wrong.
Sure, handmade lasagna still exists, but the world looks nothing like it did 200 years ago.
I don't slaughter an animal, I buy a cut of meat.
It's very rare I make pasta, I rehydrate dried husks.
The cheese comes in some kind of jar or package. The vegetables come from a store.
This has been the general move in applications too. I see companies with very large programs that are just huge sets of node modules joined together with a small amount of code.
That's an interesting thought; people "taking responsibility" as a form of labour, for machines that stole their lunch. Probably be around zero applicants for that job.
Responsibility is a complex quality needing capacity and competence. Right now even the manufacturers of "AI" are unable to assert much about behaviour.
Where "responsibility" exists around robotics and ML it will more likely be a blunt legal instrument tied to ownership, like owning a dangerous animal.
To interpret what AI models themselves are actually doing, researchers employ AI models as well.
I don’t think it has anything to do with competition or being left behind.
I think the era of big tech paying fat stacks to a rather larger number of technical staff will start to wane as well. Better hope you have top AI paper publications and deep experience with all parts of using LLMs/whatever future models there are, because if not, you’ll be in for a world of pain if you got used to cushy tech work and think it’s inevitable in a world where AI is advancing so fast.
If LLMs got this good, the brick wall these orgs will hit is what will really ruffle feathers. Leadership will have to be replaced by their technical workers in order for the company to continue existing. There's simply not enough information in the very high level plain english requirements they're used to thinking about. From a theoretical and practical perspective, you very likely cannot feed that half-assed junk to any LLM no matter how advanced and expect useful results. This is very much already the case human-to-human for all of history.
Either that or nothing happens, which is the current state of things. Writing code is not even 10% of the job.
Why don't you think that a sufficiently advanced AI can do the same as what technical humans do today with vague directions from managers?
AI car won't ever reach its destination in my city. Because you need to actively break the rules few times if you want to drive to the destination. There's a stream of cars and you need to merge into it. You don't have an advantage, so you need to wait until this stream of cars will end. However you can wait for that for hours. In reality you act aggressively and someone will allow you to join. AI will not do that. Every driver does that all the time.
So when AI will try to integrate into human society, it'll hit the same issues. You sent mail to manager and this mail got lost because manager does not feel like answering it. You need to seek him, you need to face him and ask your question, so he has nowhere to run. AI does not have physical presence, neither he have aggression necessary for this. He'll just helplessly send emails around, moving right into spam.
I can give a vague, poorly written, poorly spelled request to the free version of ChatGPT and it still gives me a correct response.
As correct as usual at least (85-95%), but that's a different problem.
There's gonna be a lot of context implied in project docs based on previous projects and the LLM won't ask hard questions back to management during the planning process. It will just happily return naive answers from its Turing tarpit.
No offense intended to anyone, but we already see this when there are other communication problems due to language barrier or too many people in a big game of corporate telephone. An LLM necessarily makes that problem worse.
Previous projects can be fed into LLMs either by context window (those are getting huge now) or fine tuning… but of course it's not a magic wand like some expect it to be.
People keep being disappointed it's not as smart as a human, but everyone should look how broad it is and ask themselves: if it were as good as a human, why would companies still want to employ you? What skills do you have which can't be described adequately in writing?
I see the threat right now to low-paid writing gigs. I’m sure there’s a whole stratum of those they have wiped out, but I also know real live humans still doing that kind of work.
What developers may use in five years is a better version of Copilot trained on existing code bases. They will let developers do more in the time they have, not replace them. Open source software has not put us all out of jobs. I foresee the waning of Big Tech for other reasons.
And they won't right up until they do. Reason why is that…
> The models are expensive to run both in compute and expertise.
…doesn't extend to the one cost that matters: money.
Imagine a future AI that beats graduates and not just students. If it costs as much per line of code as 1000 gpt-4-1106-preview[0] tokens, the cost of rewriting all of Red Hat Linux 7.1 from scratch[1] is less than 1 million USD.
[0] $0.03 / 1K tokens
Why does it feel like people think this is a useful or interesting whine? OK you brute-force solved tic-tac-toe, you built an AI. Congratulations, everyone recognises the dawn of Artificial Intelligence - and truly, enumerating all states of tic-tac-toe is all we ever dreamed of, all we could want, it's really all there is to intelligence. The term "AI" will never ever be used to mean anything else.
The term "AI" means (solving the first problem that was ever suggested to be AI decades ago) - how is that a better state of the world? What has anyone gained from "not moving the goalpoasts"?
https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
Is this just some performative grousing or do you really think what has been developed to date is “artificial intelligence”?
These comments all conveniently fail to define their author’s goalposts that apparently have been reached or surpassed. What were yours?
Notice how nobody is talking about the Turing Test anymore now that it's either already been passed or is very damn close? We can argue back and forth about whether the real stupidity was the earlier expectation that the Turing Test was a useful threshold for AI, but it's impossible to claim that it wasn't a somewhat common and well-known one, so that goalpost really has been moved in a very dramatic way (or rather, removed altogether and replaced with nothing in particular other than a vague "I'll know it when I see it", in most cases).
But it is also true that numerous ground techniques are issue of the field of AI and generally called AI as they come out. It makes for good press. And that too was silly.
For actually already done: Actually believable chat-bots? Summarizers and question answerers? Generative text and graphics actually usable for generation of text, graphics and (mostly) photo-realistic renderings? Architecture brainstorming? (And logos, etc.) Kinda working self-driving cars? New Go playing strategies? A super-human Go champion? AI is on a roll these days.
That's not counting the more proprietary and discreet applications being already used all over the place. I fully expect there are already several.
It's like arguing that when a child rides a bike with stabilisers that they have "learned to ride a bike" and then complaining anytime someone suggests the child learns to ride without stabilisers, because they "already learned to ride a bike" and now that's moving the goalposts - but you still want them to learn without stabilisers, presumably, you're just complaining about the term used to describe it, for ... apparently no reason or benefit whatsoever?
- Passing the Turing test (for some measure of that) is a huge achievement - Poopoo-ing it is unuseful.
- Moving the bar from chat-bot Turing test to wwwaaayyy over there at super-human intelligence is unuseful. There are lots of valuable steps in between.
- There are many valuable steps before super-human intelligence.
- Most humans are nowhere near super-human intelligence. They are still "intelligent" enough for significant effects on the world as well as day-to-day grind.
- You can get plenty of sci-fi-level results without super-human intelligence
- We have already achieved AGI - Artificial General Intelligence because plenty of humans operate "just fine" in the world with bog-standard intelligence and for the ones limited to a keyboard roughly equivalently to an LLM-based chatbot. Top of the line AGI OR top of the line intelligence is not necessary to massively change the world.
- Incremental and bonus goals are a great thing! You are right!
- Many humans will fight hard to reserve the term intelligence to wet stuff. That's unuseful.
- Just because "it's done" doesn't mean all of a sudden that Turing test was not a good test.
About the only point of arguing it is if you personally developed Deep Blue to beat Gary Kasparov at chess, when "beating a human grandmaster at chess" would definitely(tm) be AI, and now you feel hacked off that your personal or team recognition has been trivialised by moving goalposts and you've missed out on fame and a place in history. But I'm thinking it can't possibly be that all the people rambling about moving goalposts could be in that position and not mention it.
These things are... physically possible, but have WBE and uploads as a hard requirement. Those are going to affect a hell of a lot of things more than the drug industry!
Amusingly, machine-phase nanotechology and blood nanobots would be easier to evaluate, since simple cell-level mechanical interventions (reading surface proteins on cancer cells and chopping them up, say) will have fewer interactions than a small molecule that diffuses into every cell in the body.
Humans aren't capable of doing this, but still make useful drug discoveries. AI can be empowered to conduct research in the real world, it doesn't need to simulate everything.
This is how the human doctors who have cured things in the past have done it, is it?
The way this is going to work, when it happens, is that you'll ask the AI for a cure and it will give you a hundred candidates. A human doctor will look at the list and throw half of them out because they're toxic, several of the remainder will be excluded by animal trials, the few remaining will proceed to human clinical trials and one of them will actually work.
The rate limiting step isn't "thinking up molecules." The University of Bern enumerated all possible molecules composed only of hydrogen, carbon, nitrogen, oxygen, sulfur and chlorine, up to 17 atoms. That produced 166 billion molecules. https://pubs.acs.org/doi/10.1021/ci300415d There are commercial drugs considerably larger than that. We've got molecular structures out the nose. There is no shortage of molecules.
The problem is the clinical trial. Putting drugs in humans and seeing what they do. That's the part that takes years and tens of millions of dollars. Using AI for anything else is like saying Microsoft Powerpoint accelerated drug development. Sure, it made presentations easier, but did it do anything for the problem of putting chemicals in people?
Which is useless, because you can't run 166 billion clinical trials.
But you could run half a dozen if there's a strong chance one of them will be a success. Filtering the list down to 100 molecules from 166 billion, some of which can be further eliminated by human evaluation without the expense of clinical trials, is actually useful.
You still ultimately have to do the clinical trial, because there is no substitute for empiricism.
> That's the part that takes years and tens of millions of dollars.
It doesn't matter if it takes tens of millions of dollars if the result is a billion dollar drug.
Clinical trials only start after about five years of research and development. While they do represent a large part of the budget (even in the hundreds of millions of dollars), there are countless of other necessary steps before, during, and after trials to ensure that drugs are both safe and effective. The problem is that we still don't understand how most of these molecules behave in the body, and how we can produce them reliably and efficiently enough, which brings me to the next point:
> [...] but did it do anything for the problem of putting chemicals in people?
Yes, there are plenty of problems that AI and computational chemistry already help with in the pharmaceutical industry, including predicting solubility, stability, crystallization, granulation, toxicity, pharmacokinetics, developing the formulation, optimizing and scaling up both the synthesis and production process, developing appropriate techniques for quality control, and so on.
In all these cases and more, AI can help reduce the amount of experiments that need to be done in the lab, which require highly specialized equipment, personnel, and a lot of time. Oh and design of experiments is also a very important topic, again aiming at reducing the amount of lab time needed.
Admittedly, most of these things aim at ensuring that we do not put the wrong chemical in people, but they do represent most of the R&D effort spent in pharma, and reducing everything to clinical trials is not correct. There is a very wide gap between "AI will design drugs entirely on its own" and "AI is useless".
Google is already doing something like this: https://arstechnica.com/ai/2023/11/googles-deepmind-finds-2-...
This is great if you want to use well understood pathways or make new drugs that you can then patent and mark up.
New pathways are gonna require feeding data into these models in the first place. Your not getting ozempic out of ML without doing the ground work first: https://globalnews.ca/news/9793403/ozempic-canada-scientist-...
Sure, but a lot of the ground work has already been done, or is susceptible to simulation. They're getting a lot of results out of simulating protein folding and things like that.
Also, while not medicine focused, Google’s GNOME project results announced a few weeks ago was pretty remarkable. They discovered more theoretical new materials using their ML approach than the rest of human history combined, and they are already confirming many of the results in laboratory settings. That has the potential to be a revolution in limitless scientific and engineering applications.
AlphaFold was a nice surprise when it happened, too.