Does anyone know what GPUs the Qwen team has access to to be able to train these models? They can't be Nvidia right?
https://www.tomshardware.com/news/nvidia-reportedly-creating...
Those cards ship with 24GB of VRAM but supposedly there's companies doing PCB rework to upgrade them to 48GB:
https://videocardz.com/newz/nvidia-geforce-rtx-4090d-with-48...
Assuming the regular SKUs aren't making it into China anyway through back channels...
If you're training on more than one GPU, the speed at which you can exchange data between them suddenly becomes your bottleneck. To alleviate that problem, you need extremely fast, direct GPU-to-GPU "interconnect", something like NV Link for example, and consumer GPUs don't provide that.
Even if you could train on a single GPU, you probably wouldn't want to, because of the sheer amount of time that would take.
Those are real questions and not argumentative questions.
What does this mean? The sanctions are very specific on what can't be sold, so the spirit is to sell anything up to that limit.
25% of Nvidia revenue comes from the tiny country of Singapore. You think Nvidia is asking why? (Answer: they aren’t)
About 15% or $2.7 billion of Nvidia's revenue for the quarter ended October came from Singapore, a U.S. Securities and Exchange Commission filing showed. Revenue coming from Singapore in the third quarter jumped 404.1% from the $562 million in revenue recorded in the same period a year ago.
Then there was a second round of sanction that bans H800, A800, and all the way to much weaker cards like A6000 and 4090. So NVIDIA released H20 for China. H20 is an especially interesting card because it has weaker compute but larger vram (96 GB instead of the typical 80 GB for H100).
And of course they could have smuggled some more H100s.
However, neither access to the chips via cloud compute providers or Chinese nationals working in the US or other countries on clusters powered by the chips is restricted.