The AI Research SuperCluster
ai.facebook.com
ai.facebook.com
.... Oh wait ....
We will not be able to control an AGI any more than we are able to rein in Meta. Unless the humans in charge are held responsible for the actions of the organization they control, there is little hope we will.
Although, they are definitely a good idea considering the data source.
1) All AI is based in search (esp. non-convex, where heuristics are insufficient to provide a global convex solution), and thus is inevitably implemented using iteration, driven locally by gradient-pursuit and globally by... ways to efficiently gather information to optimize the loss function that measures how well that info gain is being refined and exploited.
2) Search that is inherently non-convex and inefficient requires as much compute power as possible, i.e. using supercomputers.
3) All supercomputer-based solutions to non-convex problems are implemented iteratively, where results are improved not using closed-form math or complete info, but by incremental optimization of partial results that aggregate with the iterations, like repeated stochastic gradient descent that creates and enhances 'resonant' clusters of 'neurons'.
4) The only form of supercomputing that has proven to scale up at anywhere near indefinitely is data-parallelism (a dataflow-specific form of SIMD) -- where the search space is spread as evenly (and naively) as possible across as many processing elements as possible.
5) Vector processing hardware like GPUs implement data-parallelism as well as any HPC architecture yet devised.
Thus, I believe that AI is stuck with GPUs, or equivalent meshes of vector processors, indefinitely.
Quantum computers are not magic and they are by no means general-purpose, but if your search mechanism matches the kind of search quantum computers are good at, they can greatly exceed the speed of GPUs.
If so, then no, using randomness to disrupt search only resets the search to continue at a different (random) spot in the global search space. That kind of approach certainly can escape local minima, but it doesn't diminish the amount of total time/effort required to do (non-convex) global search.
I think the long history of our attempting to use randomness to improve search has shown that all naive approaches inevitably fail unless the number of processors available can be scaled up to 'equal' the number of decisions required to explore the entire search space (as quantum computing seeks to do). But maybe the formal resolution of that question will come only after we prove P != NP.
The lottery tickets are random but they are basically - as I understand - smaller graphs that have the same strength as a larger graph.
It's tough to come up with a new architecture that can have an advantage on current and future models, at least from a peak perf point of view, from a perf/watt for example instead the scaled-up Apple GPUs seem to show new interesting properties. But the Graphcore architecture is quite interesting, being able to act somehow both as a SIMD machine and a task-parallel machine.
> When RSC is complete, the InfiniBand network fabric will connect 16,000 GPUs as endpoints
Microsoft's
> The supercomputer developed for OpenAI is a single system with more than 285,000 CPU cores, 10,000 GPUs and 400 gigabits per second of network connectivity for each GPU server
Is it hardward-fault tolerant? Curious how well this will work otherwise as it scales.
How can they train ML models with encrypted data?
TPUs are also just catching up on sparse workloads.
* k80 - 2014, and this really was not that great of a chip.
* v100 in 2017, and I'd consider this the first "built from the ground up" ML chip.
* A100 in late 2020, with the first major cloud general availability being in 2021.
Even when new chips come out the old ones are still usable- you can rent k80s for fairly cheap on all cloud providers, and have kept a surprising amount of their resale value. The v100s are also very much still in demand.
The A100 is also an amazing system- the new nvswitch archetecture means the A100s work together far far better than their V100 counterparts. I was part of an upgrade project setting up an A100 cluster with infiniband and it really is amazing how well these chips work together. That communication barrier was a pretty obvious next step though (the k80s had crap intergpu communication, the v100s introduced the nvlink, and the nvswitch was the obvious way to go). There isn't an obvious next step, and I imagine the A100s to be the standard for at least the next four years (with lots of continued use after that).
The T4 though isn't a "big part", but for people who fit within its envelope, it's a huge win (since it's cost is so much lower). A lot of deep learning folks had built out Turing-based workstations in that time period, and I think they're still reasonable value for money.
1. How much benefit will we get? 2. Will this benefit be higher than the cost of purchasing this right now? 3. What other alternatives will satisfy our needs?
For most of these solutions, the answers are:
1. A lot. We need computing power to perform these analysis faster and to be competitive. 2. Yes, as continuing to innovate in this space will keep us competitive and give our scientists the resources to remain productive. 3. AWS/GCP/Azure are alternatives sure, but then (the rate at which Meta probably uses these resources) it probably cost them less to build this out than to pay AWS/GCP/Azure for access to these hardware.
Granted it's probably not at this scale, but it gives you access to a ton of resources.
This makes a huge difference when using more than a single machine. I've done the math and purchased the machines at a previous company- assuming you aren't leaving the machines idle most of the time you save a considerable amount of money and get a lot better performance when building your own cluster.
I always wonder about performance on these clusters. Back in MY day, I'd wait a week or more for results from my jobs, and immediately resubmit to wait two weeks in a queue for another week of runtime and do lots of data processing in the downtime. Then, I moved to cloud and decided on "what can I afford to do overnight" (IE, I set my time to result to be about 12 hours). I have a hard time justifying additional hardware to get results in 10 minutes versus a day, it seems like at that point you're just using it to get fast cycle times on new ideas, but who has new ideas every 10 minutes?
This is the beauty of the new nvswitch chips and the infiniband networks instead of ethernet. Anyone who is doing this is setting up a fully switched high bandwidth infiniband network with ridiculous traffic between them. Nvidia purchased Mellanox a year or two ago- combine that with the ridiculously awesome nvswitch in the A100 dgx machines and there's a huge jump in cross chip traffic ability. At the same time though a decent mellanox router is probably going to set you back $30k.
Better to find algorithms that need less communication, than to make faster computers that allow you to write algorithms that needs lots of communication. Otherwise you'll always pay $$$ to reach peak GPU performance.
I mentioned in another comment that the GPU generation is roughly three years between major architecture upgrades- this has held true for a bit now, and that time may even stretch out a little. When the average company builds one of these clusters it's safe to assume they'll either run it for three years or sell it back for some return.
Going with the cloud and assuming you don't commit to several years (losing that adaptability) the yearly cost of a p4 is $287,087. Over three years that's $861,261 to run a single machine. For about $450k you can build out a solid two machine (16gpu) cluster (including infiniband networking gear and a solid NAS) that will easily last three years. There are datacenters which specialize in this and companies that can manage these machines. If you don't have the cash up front you can lease them on good terms and your yearly bill will still be much lower than AWS.
Model training is basically the one use case where I'm really willing to purchase equipment instead of using the cloud. The money it saves is enough to hire one or two more staff members, and the maintenance is shockingly low if you get it setup right to start.
The CPU/GPU part of training is only part of the much larger ecosystem, which is what you're paying to be in when you use AWS.
> you save a considerable amount of money and get a lot better performance when building your own cluster.
This heavily depends on how much benefit you get from improving GPU performance from each generation. A lot of people assume 3/4-yr TCO. If you instead "rent" for 1 year at a time, you've been getting >2x benefits per generation lately.
Most folks also measure "occupancy" for clusters like this rather than "utilization". That is, if a job is using 128 "GPUs" that counts as 128 in use. But that ignores that many jobs might have been just fine with T4s (which are a lot cheaper) versus A100s. (Depends a lot on the model, the I/O, etc.) Once you've bought a physical cluster, you're kind of stuck with that configuration (for better or worse).
tl;dr: It's not just about "idle".
At the same time it's hard to understate how different the prices are here. Our break even point for using on prem compared to AWS was about nine months. After that we saved money for the rest of that hardwares lifetime.
I definitely agree that people shouldn't just rush out and buy these without benchmarking and examining their usecase. The cloud is really good for this! At the same time though I have yet to see any cloud provider with anything even approaching the interlink I can get on prem and that means, so it's basically impossible to get the same performance out of the cloud as it is on prem right now.
At list price or a moderate discount. The folks at this scale aren't paying that :).
It is really peanuts for US government or EU.