At this point the M3 Max in a MacBook Pro with 96/128GB V/RAM needs to be benched against various Nvidia cards like 4090 & the H100 for $/performance numbers
Its even sadder that AMD/Intel have failed to take advantage of the price gouging. They could have released affordable 32GB-48GB cards with (afaik) no silicon changes.
The 7900 XTX is still somewhat reasonable where you can get it to work.
And the iGPUs are still pretty small.
We will see how Meteor Lake and Battlemage shake out, but I don't like the delay rumors.
If they intend to release an AI card, I bet they'll charge through the nose for it, just like Nvidia does.
But what they can do is double up the GDDR chips (which is what the 48GB pro cards do), kinda like running 2 DIMMS per channel on desktops. This requires a somewhat more expensive PCB and support from the memory controller, but AMD already supports this for sure, and I would be shocked if the Intel A770 die can't do it.
Impressive it works at all but the difference is night and day compared to top of the line (gamer card) Nvidia. Like SDXL <7 seconds a generation vs 60+ seconds.
(Assuming both machines are running the same model. I see a lotta crazy numbers thrown around based on comparing very different quantizations)
2 x 4090 is more like 38 tokens per second.
But it's harder than that. As the 2 x 4090 is using https://llm.mlc.ai/ which is highly optimized so perhaps the Mac Studio inference could be better when tuned.
Then a lot of production inference engines use continous batching, which means they perform really well when processing multiple prompts at a time.
As of the latest 8 bit cache commit, I can manage 2.6bpw with seemingly good quality.
For training, Apple has no CUDA, so you're pretty much dead in the water, unless you're specifically interested in e.g. local fine-tuning (for e.g. privacy or start-up strategy of "use the user's compute").
For inference, 4090 and H100 are not designed for throughput per $. Nvidia GPUs like the A40 or T40 are tuned for cloud inference, but at the end of the day your product is either running inference server-side (so play your own CAPEX vs OPEX, maybe even try Jetson) or user-side and so the user chooses the hardware not you.
I mean maybe there are Wallstreet analysts or people at Qualcomm/Broadcom who want an Apple vs Nvidia shootout to see who is getting the most of their silicon, but for the product space the comparison is basically irrelevant. Which is likely what Apple and Nvidia want right now.