2. We're working on our second generation chip. I don't know how much SRAM it has exactly but we don't need to increase the SRAM to get efficient scaling. Our system is deterministic, which means no need for waiting or queuing anywhere, and we can have very low latency interconnect between cards.
3. Yeah absolutely, see this video of a live demo on CNN!
Follow up (noob) question: Are you using a KV cache? That would significantly increase your memory requirements. Or are you forwarding the whole prompt for each auto-regressive pass?
I think currently 1. Unlike with graphics processors, which really need data parallelism to get good throughput, our LPU architecture allows us to deliver good throughput even at batch size 1.
At that price 568 chips would be $11.7M
> Accelerator Cards GroqCard low latency AI/ML Inference PCIe accelerator card with single GroqChip
If they can get costs down and put more dies into each card then it'll be business/consumer friendly.
Let's see if they can scale production.
Also, where tf is the next coral chip, alphabet been slacking hard.
Fortunately LLMs and hard work of clever peeps running em on commodity hardware are starting to make this possible anyway.
Because Google Home/Assistant just seems to keep getting dumber and dumber...
We achieve low latency by basically being a software-defined architecture. Our functional units operate completely orthoganal to each other. We don't have to batch in order to achieve parallelism and the system behaviour is completely deterministic, so we can schedule all operations precisely.
https://wow.groq.com/wp-content/uploads/2023/05/GroqISCAPape...