The capacity is there, might as well use it.
That said, if you didn't have said rack, I'm not so sure it would be worth it to even make a purchase order. Sure things outside of your control may break when you are using a cloud. But guess what, things outside your control will also break, on-prem. Particularly hardware, and network connectivity. There is no way your networking can be better than, say, GCP's own networking, or that you can deploy redundant workloads across availability zones (or even regions!) yourself.
By the time a purchase order for a new server can arrive, we can have a production-ready system running, with redundancy across availability zones, automatic failover, CDNs, backups, the works.
Basically, I don't care if someone knocks out power in my block, if someone cuts a network cable, or even if a machine goes up in flames.
One thing I would say is: even if you are very happy with your current setup, if you have some time to automate a similar setup on the cloud (keyword: automate), then I would suggest doing just that, and offload backups to the cloud too. Even if only as a business continuity thing.
My business is mainly deep learning R&D. Current cloud GPU, networking, and storage pricing gives me ulcers given my compute needs and the size of my datasets.
I do run my website in cloud, with redundancy and all that. I also use cloud storage for backup, and for K8S registry. If I was selling e.g. inference services, I'd be running them in cloud too (passing the costs onto the clients). Most of my local workloads could easily be shifted right across to any decent K8S provider.
But the fact is, my lone rack has been humming along with zero unscheduled downtime for 3 years now. I can count several global outages in each of the three major cloud services in this timeframe (most of them during US work hours, BTW), so I'm inarguably better off with the setup I have now than I would be if I moved it all to the cloud. Not to mention it already paid for itself several times over even though I burn through several hundred dollars a month in electricity.
Also, if you're sourcing components yourself or from a third party that isn't shipping specific configurations that have been tested by hundreds or even hundreds of thousands of times in other installations (i.e. not Dell), then be prepared for the possibility that you'll run into weird errors from the specific combination of mainboard, CPU, memory, NIC, RAID, etc used in a system that can be extremely painful and time consuming to to diagnose for RMA. I mean, if you're lucky, that will exhibit itself as an a crash that happens early and is easy to reproduce.
At this point, I see the premium you pay to large name providers like Dell/HP/etc as not just insurance that they will quickly accept and replace any possibly faulty hardware but also as the fee they charge for ironing out all the kinks of a platform so you don't have to worry about them for the most part.
It's totally possible to build your own hardware for cheap, but I no longer have the will to do so. To me, cloud services are like that, just to a lesser degree. I can handle the hardware locally (even if not custom built), but I'm finding less of a benefit to doing so as I get better at dynamically handling load (which has benefits in areas besides cost as well).
I have just checked the pricing of what my approximate (GPUs are different in my case) setup would cost to run on GCP, and it totals up to $40K a month. Not a typo, I did not add any extra zeros there. That's not including storage or networking costs, and _including_ the sustained use discount. That's nuts.
> it totals up to $40K a month
Yeah, it sounds like your use case is bad for cloud (not that I'm an expert), especially if you're able to get that all in one rack. It's a lot more useful if you have different layers that may need to grow at different speeds. For example, a front-end load balancing set of systems, a group of web servers to aggregate some back-end services for page display, different service clusters, and a set of databases (read-only replicas and master-master replication, etc). Both being able to scale some systems vertically (just allocate more RAM and/ro CPU to it) and other horizontally (just add more servers to a load balanced set) can be extremely useful, especially when the unexpected hits and you find you load is multiple times the normal amount all of a sudden and you need to provision now.
The more secure a system is designed to be, the more likely it is to treat unusual conditions as an attack and possibly perform some destructive action to thwart the assumed attacker. Think of phones configured to delete all data after X incorrect password attempts, HSMs with anti-tamper switches, etc.
I’ve always enjoyed this quote, but my problem with [the description of] this outage is the third-party dependency.
Packets can’t get from your cloud provider to downstream users of CenturyLink? That’s fair.
Your cloud provider can’t send packets to/from CenturyLink, so they nuke your database? I literally don’t understand.
Is the service described actually a third-party service that’s been white boxed? (I mean this in the most honest way possible. I do not understand the details, and I found the article surprising).
The root cause of nearly all of these screwups is that large, complex systems can't be fully understood or observed, and that a good chunk of knowledge about such systems is institutional, rather than explicit. So from time to time people _will_ make assumptions that don't match reality, and reality will punish them for it. Which is what, I strongly suspect, happened in this case.
It reminds me of a friend who wonders why his parents don't just install Ubuntu because windows is so awful.
Don't get me wrong, I use cloud (GCP, if you must know) too, and if my business grew massively, I'd probably use it more. But frankly I'm more satisfied with my own "on prem" solution. Single rack which basically pays for itself every 3 months or so in cloud costs, what's not to like?
Yes a major outage that’s no fault of my own could take down my site - and that isn’t great. On the other hand, wouldn’t you trust them more than yourself? Just like you trust a bank more than yourself to keep your money safe? I’m sure the average major cloud service has far better uptime than the average on prem solution. At what point does it become hubris to spin up your own on prem solution?
Nope.
>> versus using a bank
If my bank charged 25% of my balance every month, I'd very quickly choose the mattress. :-) But I do recognize that my needs are pretty uncommon.