Groq CEO: 'We No Longer Sell Hardware'
eetimes.com
eetimes.com
I just benchmarked some perf for some of my larger context window queries last week and groq's API took 1.6 seconds versus 1.8 to 2.2 for OpenAI GPT-3.5-turbo. So, it wasn't much faster. I almost emailed their support to see if I was doing something wrong. Would love to hear any details about your workload or the complexity of your queries.
In most of the cases, overall response time is mostly dominated by output as it is ~100x slower per token than input.
I believe certain companies would kill for 20% performance improvements on their main product.
I’m not convinced latency matters as much as groqs material tries to claim it does.
Let alone 'chat' use cases, but holding a reponse up for N*1.2 longer than it could holds all sorts of other resources up/down stream.
If I was using them to process far more text, e.g. summarise long documents, or if I was using it as an inline editing assistant, then I'd care more about the speed.
Streaming a response from a chatbot is only one use-case of LLMs.
I would argue the most interesting applications do not fall into this category.
…not yet anyway. Fast moving area, lots of blue water outside the chat interface.
Groq model shines at latency, not at the other two.
For example, if you're a game company and you want to use LLMs so your players can converse with nonplayer characters in natural language, replacing a multiple-choice conversation tree - you'd want that to be low latency, and you'd want it to be cheap.
(All the sudden having nightmares of getting billed for the conversations I have in the single player game I happen to be enjoying...)
If there is a future with this idea, its gotta be just shipping the LLM with game right?
That might be a nice application for this library of mine: https://github.com/Const-me/Cgml/
That’s an open source Mistral ML model implementation which runs on GPUs (all of them, not just nVidia), takes 4.5GB on disk, uses under 6GB of VRAM, and optimized for interactive single-user use case. Probably fast enough for that application.
You wouldn’t want in-game dialogues with the original model though. Game developers would need to finetune, retrain and/or do something else with these weights and/or my implementation.
Small, bounded conversations, with problematic lines trimmed over time, striking a balance between possibility and self-contradiction.
I could see it working really well in a Mass Effect-type game.
> If there is a future with this idea, its gotta be just shipping the LLM with game right?
Depends how high you can let your GPU requirements get :)
EDIT: Watching the videos, I am more and more confused by why this is even desirable. The complexity of dialogue in a game, it seems, needs to match the complexity of the more general possibilities and actions you can undertake in the game itself. Without that, it all just feels like you are in a little chatbot sandbox within the game, even if the dialogue is perfectly "in character." It all seems to feel less immersive with the LLMs.
They're selling dreams and aspirations, and those are what's driving the funding.
OpenAI and Anthropic's APIs are obviously not latency-driven. Same with comparable LLM API resellers like Azure. Most people are likely not expecting tight latency SLOs there. That said, chat experiences (esp. voice ones) would probably be even more valuable if they could react in "human time" instead of with few seconds delay.
Integrating specialized hardware that can shave inference to fractions of a second seems like something that could be useful in a variety of latency-sensitive opportunities. Especially if this allows larger language models to be used where traditionally they were too slow.
Reducing latency doesn't automatically translate to winning the market or even increased revenue. There are tons of other variables such as functionality, marketing, back-office sales deals and partnerships. Lots of times, users can't even tell which service is objectively better (even though you and I have the know how and tools to measure and better know reality).
Unfortunately the technical angle is only one piece of the puzzle.
they probably bought NVDA stock :)
My guess is that they realized that just selling hardware is a lot harder than running it themselves. Deploying this level of compute is non-trivial, with very high rates of failure, as well as huge supply chain issues. If you have to sell the hardware and support people buying it, that is a world of trouble.
> no-one wants to take the risk of buying a whole bunch of hardware
I do!
Nobody has stated it yet, but this is probably great news for tenstorrent.
Disclosure: building a cloud compute provider starting with AMD MI300x, and eventually any other high end hardware that our customers are asking for.
Semi analysis did some cost estimates, and I did some but you’re likely paying somewhere in the 12 million dollar range for the equipment to serve a single query using llama-70b. Compare that to a couple of gpus, and it’s easy to see why they are struggling to sell hardware, they can’t scale down.
Since they didn’t use hbm, you need to stich enough cards together to get the memory to hold your model. It takes a lot of 256mb cards to get to 64gb, and there isn’t a good way to try the tech out since a single rack really can’t serve an LLM.
The cloud provider path sounds riskier since that’s two capital intensive businesses, chip design and production and running a cloud service provider.
It does seem like an odd move in that case. I liken this to a company like Bitmain. Why sell the miners when you could just run them yourselves? Well, fact is that they do both. But in this case, Groq is turning off the sales. Who knows, maybe it just ends up being a temporary thing until they can sort all of the pieces out.
Since then, one of the co-founders blocked me on Twitter for pointing out that despite their claims, they were not the first MI300x to production. Neither were we, ElioVP gets that trophy, then Lamini, then GigaIO. Making us 4th and them 5th. I could go on and on with weird stuff I've seen them do, but it just isn't productive here.
Anyway, I think we have some overlap since we both are one of the few startups on the planet that actually has MI300x. But beyond that, I believe strongly that this space is large enough for multiple players and I don't see a need to be weird with each other. Apparently, I'm not on the same page though.
¯\_(ツ)_/¯
I'm not trying to point fingers, I'm just focused on building a sustainable business and listening to my customers needs. The only way I can do that is by communicating with everyone around me as clearly and openly as I can. All our customers will know exactly where they stand, at all times.
I post a lot of open information on r/AMD_Stock and the feedback that I've gotten there has been exceptional. People are excited to see if AMD can claw back a bit of the market. For the safety and success of AI, we don't need team blue vs. team red, we need everyone to work towards having as many options as possible.
This is one way that I think we are going to differentiate ourselves. We won't just have MI300x, we will have every best-of-the-best chunk of hardware that we can get our hands on. No longer will super computers be tied up behind govt/edu grants. We want to democratize it. It has long been a goal of mine to build a super computer, and here is my chance. I'm excited.
One thing that sets us apart is that my co-founder and I have a ton of experience deploying, managing and optimizing 150,000 AMD GPUs and 20PB+ of storage. We did it ourselves, all through covid and all of the supply chain issues. I'm not sure many others have done that and this is something that we are well versed at doing.
I'm also seeing my competitors hiring a ton, while we are staying lean and mean with a very small team. I'd rather automate everything we deploy and focus all of our investors money on buying compute. We also have a pool of previous people we can hire from, which I think is quite an advantage over blanket hiring.
What is the difference between this and having to sell the cloud access and supporting the people who buy a subscription?
Margins.
Pricing for cloud compute is much higher and servicing and management for the provider is much cheaper.
If I sold hardware directly, then I'm often on the hook for support contracts which can get pricy with hardware and distract from shipping future facing product features, as customers who purchase directly have longer upgrade windows due to logistical overhead.
It also isn't just the hardware capex, it is everything involved under the covers. Market pricing also factors into that as well. This is something I've struggled with myself quite a bit. I know all of my costs and what I'd like to charge, but because my offering is so brand new, until my competitors announced their pricing, I wasn't sure what the market would tolerate.
Selling hardware directly is hard for exactly what you state though. Service contracts are a pain in the butt. All of this latest AI hardware has high rates of failures too. Up until recently with AMD coming to market with a great offering, the only thing people want any sort of quantity on are nvidia products. Groq probably realized that people buying 1-2 cards at a time, wasn't going to be profitable.
Yep! You explained it better than me!
> Groq probably realized that people buying 1-2 cards at a time, wasn't going to be profitable
Yep! And if would have bogged them down by slowing down R&D cycles and even fulfilling orders as they obviously are not placing orders the same size as Nvidia or AMD.
I wonder what this portends for SambaNova and other similar vendors as well.
Time will tell. This is definitely an interesting development.
They're focused on services based full stack deployments afaik.
They come in with a rack and sell you on models as well.
Knowledge/training.
If you're shipping a brand new hardware arch, exposed as raw hardware, then you're on the hook for training everyone in the world and fixing all their weird edge case uses.
I.e. are you willing to invest in Intel/AMD/Nvidia-scale QA and support?
If you're exposing a PaaS (or even IaaS), then you have some levers you can tweak / mask behind the scenes, so only your team need be experts at low-level operations.
For a fast-paced company, the latter model makes a lot more sense, at least until hardware+software stabilizes.
At least in our experience, the first line of support is from the chassis vendors. You don't go to a store and buy MI300x. You buy them from someone like SMCI/Dell, who provides the support. Of course, behind the scenes, they might be talking to AMD. Even those chassis companies often have other providers of their gear (like Exxact) as another line of defense as well.
In the case of Groq, it would have been death by 1000 cuts to have to support end users directly, especially if they are selling small quantities. It is much easier to just build data centers full of gear, maintain it yourself and then just rent the time on the hardware.
Good thing that I'm a glutton for punishment.
(also, +100 to valuing honesty and transparency)
I'll let you know once I get my hands on them again. There really isn't enough public information about them at all. So far, my friends at ElioVP [0] have published a blog post. Still with not enough detail for my taste, but I'm pretty sure he is limited by what he can talk about. Luckily, I am not.
I mention in another comment below that my current goal is to get a bunch of people to perform testing on them and then publish blog posts along with open source code. This way, we can start a repository of CI/CD tests to see how things improve with time. ROCm 6.1 is rumored to be quite an improvement.
[0] https://www.evp.cloud/post/diving-deeper-insights-from-our-l...
Do you know if anyone's tested CuPy stuff on MI300X?
By default, we are putting dual 9754's in the chassis, along with 3TB ram and 155TB nvme. A pretty beefy box. However, if you want to work with us, we can customize this to whatever customers need.
Effectively, we are the capex/opex for something that requires a lot of upfront funding and want to work with businesses that would rather focus on the software side of things.
https://www.reddit.com/r/LocalLLaMA/comments/1bpgrdf/wanted_...
I've got about a dozen people signed up. Just working through some hardware issues right now (see above about high rate of failures), and hope to have this resolved next week, so that I can get people onto them and doing their testing.
They _should_ work fine for both training and inference, but since nobody has done much in the way of public in-depth benchmarks yet... I was hoping to get people to do it for us in order to stay as unbiased as possible.
noticing now: strange that my previous comment was downvoted. Would be nice to understand what someone didn't like about what I said!
The thing that nobody talks about is that there is a high rate of failures on this high end equipment. I've heard as high as 20%, in the first month. I'm not even talking about AMD here.
If anyone thinks they can just buy some accelerators and throw them into a rack and expect them to work flawlessly... they've got some hard lessons to learn.
This will be less of an issue as we grow as we will have plenty of stock to pull from, but it is a real bummer as we are starting as a proof of concept first. We started working on this business last August, before anyone knew whether or not AMD would even change course on AI.
The good news is that we onboarded a customer the day that we announced our availability, we passed that PoC challenge with flying colors and closed significant additional funding immediately after that. Onwards and upwards, just have to roll with the punches.
And I'm not even talking 100/400G network, wonderful wonderful hardware, good luck debugging and getting all the RoCE/RDMA/GPUDirect/StorageDirect/NCCL working (already a bit of pain on nvidia, with a large installed base...).
Either you want to learn all this stuff (for reasons) or you're dumping a lot of money on fast-evolving tech.
5 yr old silicon (14 nm!!) and no hbm.
Their secret sauce seems to be an ahead-of-time compiler that statically lays out entire computation, enabling zero contention at runtime. Basically, they stamp out all non-determinism.
It's not inconceivable that's a better trade-off than leading-node and HBM requirements.
(the way I understood it => it's still cost effective at scale due to throughput increase this brings)
No doubt fast SRAM helps, but from a computation pov imho its that they've statically planned computation and eliminated all locks.
Short explainer here: https://www.youtube.com/watch?v=H77tV1KcWIE (Based on their paper).
Most important, even ignoring latency, is throughput (tokens) per $$$. And according to their own benchmark [1] (famous last words :)), they're quite cost efficient.
[1] https://www.semianalysis.com/p/groq-inference-tokenomics-spe...
So they attacked the italicized portion and simplified the hardware. Mostly by eliminating memory-layer non-determinism / using time-sync'd global memory instructions as part of the ISA(?).
This apparently reduced the difficulty of the compiler problem to something manageable (but no doubt still "fun")... and voila, performance.
EDRAM is essentially a tradeoff between SRAM and DRAM, offering much greater density at the cost of somewhat worse throughput and latency.
There were a couple of POWER cpus that used EDRAM as L3 cache, but it seems to have fallen out of favor.
For a minimum 100 wafers = 10k chips, Groq may have paid $100M = $10k/chip purely in amortizing design costs.
Chip design (software + engineer time) and fabrication setup (lithography masks) grow exponentially [1][2] with smaller nodes, e.g., maybe $100M for Groq's current 14nm chips to ~$500M for their planned 4nm tapeout. Once you reach mass production (>>1000 wafers, which have ~150 large chips each), wafers are $10k each. On top of this, it takes ~1 year to design then have prototypes made. (These same issues still exist on older slower nodes, albeit not as bad.)
This could be reduced somewhat if chip design software were cheaper and margins were lower, but maybe 20% of this cost is due to fundamental manufacturing difficulty.
(disclosure: I don't work with recent tech nodes myself; this is my best guess)
[1] https://www.semianalysis.com/p/the-dark-side-of-the-semicond... [2] https://www.extremetech.com/computing/272096-3nm-process-nod...
Think about the amount of money being dumped into "AI" at this point. If you've got the technology and people to make stuff faster/better/cheaper, finding investors to dump money into your chip making business is probably not as hard as it was 2 years ago.
Groq is making this change for other reasons than the expense of tapping out chips.
can’t comment on specifics, but imo our hardware team punches above its weight class in terms of # of people and time spent in design.
That said I think their arch is super interesting. I just think that demo was way too hype when the actual system is pretty impractical.
Operate at a loss. Get a big valuation. Cash out.
I want to use it, but it's been very unreliable. I have been using Claude 3 and thinking about together.ai with Mixtral.
First, the whole systems of chips architecture that everyone is talking about will solve for increasing overall SRAM available to keep more model state on super fast memory and avoid going to slow memory.
Secondly, anyone serious about their data (enterprises) won't be okay with making API calls to Groq. Anyone serious about their data and have a lot scale (consumer internet) won't also be okay with making expensive API calls to Groq at scale.
Their cloud is attractive only if I can use their API for experimentation toy apps to continue developing in this direction while the rest of the major industry players systems of chip architecture catches up and solves for SRAM size bottleneck and manufacturing process bottleneck, and once that's solved, I get more powerful compute for cheaper $$ to deploy on-prem.
So, this cloud strategy is short-lived. I see another pivot on the horizon.
Linked article:
If customers come with requests for high volumes of chips for very large installations, Groq will instead propose partnering on data center deployment... and yet, they're still leading the field.
I think it's a bit early to think the field is getting commoditized yet.
I think one major challenge they'll face is that their architecture is incredibly fast at running the ~10-100B parameter open-source models, but starts hitting scaling issues with state-of-the-art models. They need 10k+ chips for a GPT-4-class model, but their optical interconnect only supports a few hundred chips.
[1] https://www.zach.be/p/why-is-everybody-talking-about-groq
> what do you mean I can’t just drop a CUDA docker image in?
A hardware startup that sells cloud access to its hardware. :-)
Do you have any idea how fast Groq is? Go try it. Consistently over 400 t/s for most of the models that they support, and extremely low latency.
remember that EU -> US is ~150ms unavoidable latency, for example. then your comparison is local H100 vs Grok + 150ms latency to first token.
I said "and extremely low latency" because I know they are different. Groq's TTFT is still consistently competitive with any other provider, and lower than most of them. Here's some benchmarks: https://github.com/ray-project/llmperf-leaderboard#70b-model...
Their hardware was never for people at home, but for cloud providers.
A 7B model would then be able to run on about 60 LPUs. Even at $20,000 per card that would be only $1.2 million and I highly doubt the cost is actually that high, that's just what DigiKey says the cost of an LPU is, if you're trying to buy just one :)
Tenstorrent also looks incredibly Python-specific (as in, everything including their SMI seems mostly Python-based) which doesn't seem promising?
(It was all Python up until ~6 months ago)
EDIT: Oh, and you can buy the Grayskull cards online now, without contacting anyone.
I actually don't mind having to contact, I only mind if they won't want to sell to me due to being a non-bulk order.
> Most of the low-level pieces are in Rust
That's awesome!
> “There might need to be a new term, because by the end of next year we’re going to deploy enough LPUs that compute-wise, it’s going to be the equivalent of all the hyperscalers combined,” he said. “We already have a non-trivial portion of that.”
Really? Does anyone seriously believe they are going to be the equivalent of all hyperscalers in compute next year? (Where Meta alone is at 1 million H100 equivalents.) In the same article where they say it's too hard for them to sell chips? And when they literally don't have a setup to even accept a credit card today?
https://www.tomshardware.com/news/no-sram-scaling-implies-on...
IIRC the last big jump for SRAM density was at 7nm, so they do still have that card to play, but progress has slowed to a crawl beyond that. TSMC 3nm SRAM is barely denser than TSMC 7nm SRAM.
What? How does this make sense?
[0] https://deepmind.google/discover/blog/millions-of-new-materi...
[1] https://www.aramco.com/en/news-media/speeches/2024/leap24--r...
Groq's system was designed to run arbitrary high performance numerical workloads. In the past it has been used for a variety of scientific computation tasks, including nuclear fusion and drug discovery.
https://www.alcf.anl.gov/news/argonne-deploys-new-groq-syste...
For vanity reasons and because AI is the future (not every company acts that rationally for huge buying decisions).
Groq is still under a 30 request per minute rate-limit, which drops to 10 requests per minute if you have all day usage.
Billing has been "coming soon" this whole time, and while they've built out hype enabling features like function calling, somehow they can't setup a Stripe webhook to collect money for realistic rate limits.
They couldn't scream "we can't service the tiniest bit of our demand" any louder at this point.
_
Edit: For anyone looking for fast inference without the smoke and mirrors, I've been using Fireworks.ai in production and it's great. 200 tk/s - 300 tk/s is closer to Groq than it is to OpenAI and co.
And as a bonus they support PEFT with serverless pricing.
I just have free API access with no ability to add a credit card.
The story telling site alone averaged 27k requests a day this week, so about double what their current request limit is, and honestly not even that popular of a site.
You can't run much more than a toy project on their current rate limits.