In general I'm with you, but that here is the main benefit imho. I don't have the nerves to operate my own redundant, transparently growing and reliable storage system for my business. For use cases where privacy is important (like backups, private user data, etc), encrypting data before uploading limits the privacy leaks to metadata which might be acceptable.
I am genuinely interested in what you think about things like this:
ssh user@rsync.net du -Ahd2 some/directory
pg_dump -U postgres db | ssh user@rsync.net "dd of=db_dump"
or maybe even: ssh user@rsync.net "git clone git://github.com/freebsd/freebsd.git freebsd"
Setting aside these "rsync.net tricks", I think you should look into the 'borg'[1][2] backup tool. It has become the de facto standard for remote backups because it does everything that rsync does (efficient, changes only backups) but also produces strongly encrypted remote backup sets that only you have a key
to ...The real argument against S3 though is its price. For any but most trivial amounts of data it's terribly expensive and I use it only as internal storage for other AWS services.
One could self-host minio which is nice S3 replacement: https://github.com/minio/minio
(Assuming an already enterprise-y prosumer IS, i.e. pfSense, VLANs, ZFS storage, 'enough' CPU/RAM/network, all patched and well in good sysadmin fashion, inet-facing resources behind some reverse proxy, etc).
The first target is internal customers for a consulting dev shop — i.e. me + n first partners / founders / employees (I'll begin alone but with a goal to grow to ~3 people asap).
So website, storage w. inet-facing endpoints, security all the way (encrypt etc), and basic IT stuff for dev: e.g. private GitLab + straightforward CI/CD pipe. I figure all developers need is an endpoint to push git, and access to the automation processes.
____
Longer elaboration, FWIW (maybe not much to answer the question but I'm a bit over my head on this topic so I'd rather share too much, to get debunked where needed).
The rationale to bootstrap as self-hosted is simply that it's already paid (gear is humming as I write) and I'm “99% sure I can guarantee 99% uptime” (the full setup is split in two homes 500km apart, each behind 1Gbps symmetrical business-grade fiber, each site being in High-Availability locally— from router/firewalls to storage passing by proxies and k8s-pods).
Re certs, well in a bootstrap scenario we'd worry about those later I suppose, when there's actual demand to justify the overhead cost (first clients are likely to be extremely low-profile, looking for a "break" too). Legally, clients own their code and infra at the end of the day, we would deliver a product/tool but not host it (not an MSP; we'd typically rather work with their MSP— or recommend one —to smooth out their operation, add critical value to key pain points).
In terms of actual workloads, I'm looking at custom domain-parametized ML/DL services (think: selling very specialized neural nets that solve one problem extremely well for this one company, e.g. classifying internal docs). So I need some in-house hardware anyway to experiment when developing models (it's painful and limiting to be forced to spend at the exploratory stage, for each project).
I figure my first partners or employees would either run their stuff locally or tap into my GPUs; we'd likely pay for the big guns on AWS or GC only for actual prod training iterations (those that earn us actual money when our work is 'good enough', final drafts so to speak).
It's a bit of a maverick project¹, hoping to leverage ML in profitable ways for small operations (obviously as we accumulate clients and experience, we'd get better at nailing areas of best-value optimization). It will be a rocky road during the early phase (first two years at least). The costs can skyrocket pretty fast with ML on clouds, I'm extremely wary of such fuckups, not to mention running the ongoing risk of being cancelled or shutdown (account terminated or frozen) for whatever reasons — so better have "plan B"... which historically might have begun as plan A, i.e. self-hosting.
I've also got a long-term angle of building expertise, so I've a positive bias for on-prems: if this thing works and grows enough, we'll definitely look at running the operation in-house, because clouds cost proposition for ML is too steep to scale profitably if it's your core business, or so I hear (and before one says "but Spotify is on whatever cloud", yeah, that's not their core business, they don't sell AI but music; we're the guys who sell AI to Spotify ideally). So, in ML for now, gotta get'em GPUs and TPUs to make a mark. Thus I'd rather learn all of it (myself maybe, as a company surely) sooner rather than later. It's gonna come down to this anyway, if this maverick thing works.
1: "maverick" as of 2020, but I suspect there will be a whole market of such consulting / ML shops by 2030.
However, S3 isn't trying to replace your NAS. S3 offers many use cases where it is the smartest and most viable solution. The most common use case is as centralized block storage for an online web app, that runs on load balancers, a CDN, and distributed servers. If you run an app that is going to run on multiple servers and get load balanced between them, then uploading directly to the servers is complicated. Since a user could upload something to server A one time, and a month later they could request the file while visiting on server B, C, D, E, etc. This centralized S3 platform simplifies the block storage for these large apps.
S3 does offer more advanced features as well that many people aren't educated on. For example, S3 Lifecycle Management allows you to set up timelines so that your file uploads are available on S3 for 30 days (when users are most likely to access them), then automatically move them to reduced redundancy (reduces cost, but at lower 99.9% SLA) when the file is 30-120 days old, and after that it moves to Glacier storage (long term, taped storage, super cheap), for long term storage when it is unlikely to be retrieved but needs to be kept. You can also set up a CDN just for this block storage, with over 200 edge locations all around the world.
S3 also offers HIPAA and PCI compliance. There is also incredibly robust user management (Through AWS IAM). Plus you get 11x9's of availability (99.999999999% availability).
Im not trying to sell you on S3. It definetely isn't for everyone. There are times when I would tell people that it isn't a good fit. But if you work on enterprise apps, it gives you a lot of enterprise level features for month-to-month, (relatively) straightforward pricing, no long term commitements, etc. Which is something that doesn't generally exist in the world of enterprise software or PAAS (which is what S3 really is). So for that, S3 is a breakthrough product.
S3's biggest flaw is its price. If you are using all the enterprise features than it is arguably worth it, especially considering the 11 nines of availability. But for anyone on a non-enterprise app that doesn't need that stuff, there are far cheaper solutions out there for you. Blackblaze B2, Wasabi, Digital Ocean Spaces are three that come to mind.
Seconded on the last two in this list. I use both, for completely personal storage needs. I was lucky enough to get Wasabi's older, cheaper pricing model for the lifetime of my account, but even so it's a hugely attractive option for storing large amounts of data without worrying about local equipment and all the rest of it.
You'd like https://scaleway.com free-tier of 75GB object storage / transfers per month.
Note that those 9s are for durability
From https://aws.amazon.com/s3/faqs/#How_durable_is_Amazon_S3:
"This durability level corresponds to an average annual expected loss of 0.000000001% of objects. For example, if you store 10,000,000 objects with Amazon S3, you can on average expect to incur a loss of a single object once every 10,000 years"
I agree for the most part on cloud, but isn't "another person's computer" just a server? Serverless is more like a service running on a server that requires zero configuration or maintenance for you, just add your code and it works. Typically that means no messing with apache, nginx, no language nor framework setup/configuration, etc...
Whoa, I thought it was: Use a server, use a CDN, shutdown the server and its running serverless. Because everything is getting hitting the CDN and not the actual server.
Obviously it's on a server, but the "serverless" part comes from not having to provision and manage a server. You can put your code a a service and allow those services (S3, Lambda, API Gateway, etc...) to do the "server" work for you.
Serverless is kind of a generic term. I guess hosting sites like GoDaddy that allow you drag and drop some PHP, HTML and JS files into a CPanel type interface are technically serverless, since you're not managing a server. Put some code up there and let their platform run it for you.
For example, going back to a serverless frontend using S3 that I mentioned before. You can host a JavaScript SPA on S3 and use a service like Route 53 for DNS. Your JavaScript can then make API calls to a full-on server that you manage, say an EC2 instance with a LAMP stack or even Azure with C#/.NET. Your front end is serverless because it's just hosted on S3 and you're not maintaining a server, but your backend code would be deployed and served however you like, such as Jenkins and Kubernetes like you mentioned.
AWS lambda is a good intro to the world of serverless computing if you're not familiar with it. While you can write code on your computer and use the AWS CLI to upload it, the AWS console even has a text editor that resembles JS Fiddle. It can receive events from other AWS services and you can write JavaScript, Python, Java, Ruby, C#, Go and PowerShell to process those events and do whatever you like, such as writing a DB record or adding something to an SQS queue. The point is you're writing real production code that does not live on a server you manage.