I'd prefer the latter since I won't have to worry about maintaining hardware, but I heard the prices are pretty high and there's slowdown on some AWS GPU instances? Thoughts?
If you're just getting started and don't have a specific application in mind you should just play with some smaller problems, use Colab and/or use the $300 in GCP credits Google keeps advertising.
If you're going to be doing this for an extended amount of time, then buying hardware will be better; you can get some good deals on second hand high end GPUs on eBay.
The main risk is that you buy a bunch of gear, and then don't use it.
I think you're hitting the biggest up/down sides on the nose.
There is an innate joy to building your own rig (I'm guilty of it myself), and it does provide a fair amount of flexibility. The two main downsides that come to mind are the up front cost for parts, and the maintenance required to keep the system up to date. I've had success with a CentOS rig running nvidia-docker[1]. The annoyances mount over time though, as OS and Nvidia driver releases can be a pain to keep up to date, especially if you're working with several of the major frameworks. This is the main reason I've moved most of my ML projects to the cloud.
Cloud has many benefits including scalability and minimal maintenance. There are many options in the cloud ML space, but my current favorite leverages Google's cloud. I'll get into detail below, but the main reason is that they have several tools that build a natural progression and will grow with you as your projects mature.
First Steps:
I'd recommend checking out Colab[2][3] to get your feet wet. It includes a FREE GPU instance, and is a Jupyter based notebook, so you can easily utilize any of the python based frameworks. (1/2 of a K80, if memory serves. 12 GB) As with all free resources, there are some caveats. Colab is designed for interactive usage, so they don't encourage long running processes. However, the general consensus online is that you can get around 12 hours of training in at a time before the instance restarts itself, which is plenty of time to kick off a job and save checkpoints to cloud storage for resuming later. All in all, it's a great tool get a project off the ground. One bit of strangeness (from a software engineer's perspective): the notebooks are stored in google drive. This adds an interesting possibility for collaboration (it's in the name, right?), but leaves something to be wanted in terms of source control.
Growth:
If Colab isn't cutting it, there are a couple next steps. If you need more compute power, or have long running training jobs, the Colab UI can actually be run locally backed by your own GCE instance (with attached GPUs)[4]. If you need additional services (databases, map/reduce, etc) Datalab[5] is Colab's bigger beefier cousin, with all sorts of integrations across the Google Cloud Platform, as well as 3rd party providers.
EDIT: Link formatting
[1] https://github.com/NVIDIA/nvidia-docker
[2] https://colab.research.google.com/notebooks/welcome.ipynb#re...
[3] https://research.google.com/colaboratory/faq.html
[4] https://research.google.com/colaboratory/local-runtimes.html