Is their hardware programming interface reasonable for implementing inference of frontier models: no quantization, several tera params?
BTW, how many many params open weight frontier models have? A few teras, 100s of teras?
BTW, how many many params open weight frontier models have? A few teras, 100s of teras?
Qwen3.8-Max: 2.4T
DeepSeek V4 Pro: 1.6T
DeepSeek V4 Flash: 284B
(all are total parameter counts, not active parameters)
My personal take, with no sources: 100T sounds excessively high given they need to be able to actually serve these things on commercially available hardware. I would guess they are in the same order of magnitude as the Chinese frontier models. It's possible their edge is in RL training methods, training-time compute, and access to data (e.g. from customers' CC/Codex sessions), not in model size.
This is so hard to sort the true from the false nowadays.
If LLM becomes that good at coding, I'll have to run an open weight one locally.
I have no clue anyway.
Compare the of bedrock inference on known open source models to bedrock pricing of SOTA models. If the price of SOTA model is 10x qwen3 480B the the SOTA models is max 10x larger. Obviously anthropic Openai etc take a hefty cut from AWS for letting or run their models, so 10x more expensive model is probably only 5x heavy to run.
Like linux DMABUFs with userland hardware command ring buffers (I guess this hardware ring buffer instance would be specific to a VMID and a PASID).