Admin:
Time (and cost) it takes to install software, drivers, etc
These tasks aren't made unnecessary by cloud. Yes, with cloud, once you've done the work, it can be forever encoded into an image or container. However, the same applies to on-prem using container solutions like Docker.
Maintenance / Capitalization / Finances:
3-year? What is the useful life of this? When does it seemingly become obsolete?
With GPUs of recent history, obsolescence is not a concern in a 3-year time frame. On AWS, people are still using K80s (released in 2014!). The GTX 1080 Ti, which was released 2.5 years ago, is selling for substantially above MSRP. This may change if competition in the GPU space increases and NVIDIA loses its monopoly.
AWS will continually upgrade their hardware and you keep paying the same
True, but this concern is mitigated by slow rate of GPU obsolescence.
Hardware can be capitalized, which means you can push it to the balance sheet (for tax or valuation purposes)
Can you say a little more about this? Not sure what you're getting at.
Spending $90k instead of $184k in year 1 with the option to turn it off if you want (no longer need). This could be very valuable for a startup who wants elastic spending patterns.
This needs to be evaluated on a case-by-case basis. A poorly-capitalized start-up with an unpredictable compute workload is not a good candidate for buying on-prem. A well-capitalized start-up that consistently uses GPUs is a better candidate.
Returns, breakage, warranty in case of a hardware failure
Any reasonable hardware provider will include an option for 3-year warranty.