Bing Chat is so hungry for GPUs, Microsoft will rent them from Oracle
theregister.com
theregister.com
I'd say Google is in trouble, I mean there is a limit to how many ads you can pack on the first page of results and I'd say they've passed it a long time ago.
In that context Bing + GPT would be more of a winner I think.
That's why most implementations add a layer of Retrieval-augmented generation (RAG): https://docs.aws.amazon.com/sagemaker/latest/dg/jumpstart-fo...
In a world where everyone is desperate for GPUs, Oracle has spare GPUs to rent out to competitors.
What does that say about demand for Oracle’s cloud offering?
The actual cause here is something else: NVIDIA have been allocating GPUs according to long term corporate priorities rather than just auctioning them off and letting price signals do the work. The reason is twofold:
1. They want a diversity of customers for GPUs, not just one or two LLM mega-players that everyone else invokes via high level cloud APIs. That would put NVIDIA in a bad negotiating position.
2. They don't want GPU supply to be dominated by companies that are simultaneously designing their own silicon. Microsoft is. [1]
Oracle has been getting large allocations because they aren't designing their own AI hardware (or at least, if they are, this isn't public), and because they are in the business of selling CUDA and not hardware abstracted model access.
[1] https://www.theverge.com/2023/4/18/23687912/microsoft-athena...
Right?
Also, I'm guessing that Oracle is sufficiently desperate to wring any revenue out of their investment in cloud hardware so this might be a rare occasion where Oracle will just take the money and call it day.
Larry Ellison quote.
Ellison's opinion is given a moment later:
"And we certainly believe that that is absolutely correct, that we have to pursuit market share. Today we have more share than all of our competitors combined. And our primary goal is to improve our share, our market share."
Jack Welch over at General Electric: "If you're not one or two in the market, you don't make money. The market leaders make the money." You know, you have to have share. You have to have substantial market share in order to be profitable in most businesses. The Japanese recognize this very, very clearly. And not just the Japanese. The great business leaders in the United States are the same, whether it is the old guard--and Jack Welch forgive me for calling you the old guard--or the young guard like Bill Gates. And we certainly believe that that is absolutely correct, that we have to pursuit market share. Today we have more share than all of our competitors combined. And our primary goal is to improve our share, our market share.
Still at least you provided a source. The actual full quote is Ellison saying that he was told this story by a Japanese business executive, and when he repeated this story to a New York Times journalist they took it totally out of context and made it sound like it was him saying it. He even says he's got in trouble for repeating this story before, but it's OK here because it's on video and therefore hard to take out of context! But on HN that's no obstacle apparently.
DM: Going back to Japanese culture, one of the quintessential elements of the Japanese approach to competition has been the aggressive pursuit of market share. Was your aggressive pursuit of market share early on something you learned, or was it just the natural outcome of heads-up business?
LE: No. I think it was absolutely our primary goal. I was always influenced by Japanese business people. In fact, it was a very famous story that I've gotten in a lot of trouble for, and I'm very hesitant to mention here, but I will, because at least it's on video and it's harder to take things on video out of context. But I was in Japan and I was talking to a Japanese business executive. And he told me that the problem with America is that we just have no stomach for competition. And I was defending America. I was defending my country and my culture. I said, "What do you mean? America, at the turn of the century was the leading edgeI mean, we were the laboratory, you know, for free market economics. The industrial revolution reached its zenith in the Eastern States of North America. What do you mean?" And he says to me, "Don't tell me about things that happened almost 100 years ago. Let's talk about today. You listen to your business executives and they say things like, Well, we have great respect for our competition; the market is very large; there is room for all of us to compete."
"In Japan, you know, that would be sacrilege," he said. "In Japan, we believe our competitors are stealing the rice out of the mouths of our children. In Japan, we think anything less than 100 percent market share is not enough. In Japan, we believe it is--and this is a quote that supposedly came from Genghis Khan, but I have heard it attributed La Rochefoucauld and other people--it is not sufficient that I succeed; everyone else must fail. We believe that this is not sufficient. We must destroy our competition.
And I remember I came back with that story and told it to people at Oracle. And I remember telling a New York Times reporter this story. And the New York Times reporter had a huge picture of me, took the whole thing out of context. It said, Larry Ellison, quote, "It's not sufficient that I succeed; everyone else must fail." With a huge picture. Then I started getting hate mail. (Laughter) I mean, this story stuck with me for a long time.
Bing's market share increased by 0.1 percent? What is the point if mentioning this?
If I was on the Bing team, I’d take that as a win. This could be the start of the hockey curve uptick!
On commodity markets. GPUs are essentially a monopoly, so gluts are only really possible with outdated hardware.
My favorite eBay listing of today is a lot of 100 NVIDIA P106-100 6GB GPUs for ~$20 each: https://www.ebay.com/itm/305023042595
For those looking for the cheapest higher memory solutions, 24GB P40s are available (decent amount of VRAM, 3X the memory bandwidth, but requires server or DIY cooling, same bad FP16) or IMO the best bang/buck for hobbyists/home devs atm, used RTX 3090s are going for about $600-700 each.
(Note: if you're doing training, unless you have very high utilization/already calculated your costs, you will probably be much better off renting cloud GPUs from Vast.ai, RunPod, etc)
This is not exactly accurate. P106-100 were not specifically made for mining, unlike NVIDIA CMD Hx cards (https://www.nvidia.com/en-us/cmp/). P106-100 cards are NOT artificially throttled, but the specs are still very poor compared to modern cards: https://www.techpowerup.com/gpu-specs/p106-100.c2980
It is true that they are not worth the power costs. I merely made a point that gluts of outdated hardware is a thing even when NVIDIA is a monopoly.
Beyond that, I'm pretty sure that the Techpowerup DB is wrong on the PCIe bandwidth. The first-hand reports I've seen online says the P106's are limited to PCIe 1.1:
* https://www.videogames.ai/2019/01/18/P106-100-GPU-Machine-Le... and 2022 followup on P106-90 https://www.videogames.ai/2022/11/28/Stable-Diffusion-Mining...
* https://www.reddit.com/r/lowendgaming/comments/135fymv/bough...
* https://linustechtips.com/topic/1026742-some-information-abo...
$20 is a fair price for those that actually need exactly what the P106 provides, but the cheeky flipside argument is that having a near monopoly might make even the e-waste overpriced. :)
(Disclaimer: I work for AWS, but opinions are my own.)
(Disclaimer: I used to work there but not on them.)
HN is not a representative sample of industry. Postgresql and Go are, of course, well-used, but Java and MySQL / Oracle are still absolutely massive.
Oracle, Java, C#, MS SQL might not be flashy in tech startups and on HN, but they are absolutely dominant in the wild.
I hope all this AI money sloshing around spurs other players to make compelling GPUs.
AMD MI300
It is very hard to source the power required now.
The rest of OCI is a step down, but it's mostly good enough
You can see all alternative GPU providers: https://gpumonger.com/cloud-gpu-providers as well as the main ones.
Oracle has great GPU prices: https://gpumonger.com/cloud-gpu-provider/oracle-cloud
Everyone wants to get into AI because end users love it and are willing to pay for it.