In praise of S3
info.acloud.guru
info.acloud.guru
I sometimes have to support freelancers working in some of our Wordpress websites. Their first instinct when something is wrong on their end is asking me to run a chmod or chown command they found on Google on the whole directory. Not that it matters – we're using Docker.
Security seems to be secondary when the priority is to just deliver.
That’s different from saying “security should be an afterthought”. Security is something you should consider consciously and prioritize against your other goals.
Putting security first is kind of weird if you think about it. Imagine building a house and prioritizing the locks.
Security is always an afterthought to those who don't actually understand it.
This reminds me of how React has a dangerouslySetInnerHTML[1] API - sometimes it's the hammer you need, but the name should (hopefully?) cause pause.
[1]: https://reactjs.org/docs/dom-elements.html#dangerouslysetinn...
It'll still blow your head off.
Don't be overly discouraged :-)
I have a reliable (if a bit ugly) implementation in under 3kb of sparse PHP OOP code. Yes it took a few hours to iron out the last kinks, and one extra refactor to free up a mistakenly bound variable. It might have helped that it was the first version I implemented, thus no need to match impedance of older interface.
On the plus side, the signing is designed as self-contained, and neither implementation nor API calls spill to too many parts of request handling.
>you have to read the doc a dozen times before you can successfully implement it
Very much so - it does require a small 'eureka' moment. It's hard to put your finger on what exactly makes it hard to get right the first time around. At the very least it is written from a somewhat peculiar viewpoint.
Fortunately the primitives used are common enough to be available in typical standard libraries.
Put another way, if you look at your service's API and it's complicated, you probably don't have the abstraction right.
Our most common flow involves files stored on S3. The app will download the index, then use sampling and streaming techniques to pull from the full file. This works well because S3 supports range requests.
Where S3 falls down for us is sharing. I'm not aware of any easy way to share S3 files with email. You either have to create a signed URL (which has a max expiration), make the file public, or use IAM, which I'm pretty sure requires all users to have AWS accounts. That's a non-starter.
Google Drive is the opposite. It's better for sharing. Just drop in an email address, or share a permanent link. But it no longer supports range requests, so it's essentially walled off from the rest of the web.
What's crazy to me is that neither of these services provide a non-authenticated way to use a CLI tool to download/sync a directory tree, even if it's public. rclone is probably the best tool in this space, and it requires you to configure "remotes" ahead of time. You can't just point it at a public bucket and have it download.
I think there's still room for improvement and competition in the cloud storage space.
It is actually possible to run 'rclone' without configuring a remote ahead of time. It's relatively recent, but if you look at this discussion thread:
https://forum.rclone.org/t/can-rclone-be-run-solely-with-com...
... and the later development that was done on this ... you will see that you can run a single 'rclone' command (albeit, a very long and complicated one) that will specify everything that was in your remote configuration.
I hope this is helpful ...
"I think there's still room for improvement and competition in the cloud storage space."
We're trying[1] ... and specifically, with rclone[2] ...
I've been keeping an eye on rsync.net for while. I really like a lot of the things you're doing. I think SSH is pretty much the best way of doing remote file management that we have today. The main problems are it's hard to share stuff and it doesn't work in browsers.
Thank you - appreciated!
" think SSH is pretty much the best way of doing remote file management that we have today."
I will alert you to the issue of deep inefficiencies in SSH/SFTP over long WAN links. This problem has been solved by HPN-SSH[1] and is likely already built into your OpenSSH distribution, but I mention it here because it is not very well known ... we run into customers doing long, distant (hops) transfers over SSH that we sometimes need to step through the process of updating their ssh so they can negotiate HPN-SSH with us. It's also worth noting that you can configure ssh do use a "null" cipher which might be applicable with borg uploads, bit I digress ...
"The main problems are it's hard to share stuff and it doesn't work in browsers."
This is true - and by design. There will never be anonymous / non-credentialed access to rsync.net accounts and no rsync.net storage array will ever run an httpd. Our nmap results will always be one line long: TCP22/sshd.
Good to know, thanks!
> This is true - and by design. There will never be anonymous / non-credentialed access to rsync.net accounts and no rsync.net storage array will ever run an httpd. Our nmap results will always be one line long: TCP22/sshd.
Makes sense from a security standpoint. But I think there's a business opportunity for having cloud storage that allows you to host your files over HTTP. Imagine if google drive was your CMS. Just drag your static site over and it's live. Edit HTML files in-place. Set permissions based on path.
What's the difference between a permanent link and simply making the file public? (The email address, that's essentially requiring the person to have a Google account, so that's the same as IAM.)
Does Drive require a google account? I haven't tested that. If so that's really sad. I know you can share with non-google email addresses but maybe they have to already be logged in with a google account for it to work.
- You can create public pre-signed URLs with an unlimited expiration date using SigV2. I doubt SigV2 will stop working anytime soon.
- Or, if you want the link to provide access to anyone, why not make the object public, but put a UUID in the key name to prevent enumeration?
I like the UUID idea. Especially since you can always move the file under a new UUID to revoke previous access. Not sure if there's a way to generate a new link with GDrive.
If I could add one thing to it, it would be a way of listing a bucket to get the objects and the metadata back at once - it is another HTTP request to get the metadata otherwise.
The large object handling is pretty good (unlike, lets say the OpenStack Swift API which has 2 ways of uploading large objects both of which basically leave everything hard to the caller!). Large object handling (say objects over 5GB) is the Achilles heel of all the cloud storage systems.
The myriad ways of authenticating with S3 (which have grown like topsy over the years) are a significant complication. However that is the same for all of the Enterprisey cloud storage solutions (eg Azure blob storage).
There are quite a few re-implementation of the S3 API, CEPH, Wasabi, Minio, Alibaba cloud, etc all of which are more or less 100% compatible (though the rclone integration tests have winkled out a few small differences mostly to do with handling of characters like LF and CR).
So impressive work AWS, making an API which has stood the test of time and become a de-facto industry standard.
Also thanks for implementing sync to different storage providers. It's probably tempting to just implement the big players, like S3, but little ol' people like me appreciate the rest :)
It has been a voyage implementing so many cloud providers and keeping them working. You might be surprised how much code is tests which get run against the cloud providers every night in the integration tests!
A thing I wish they have is a way to get a consistent view of the object (similar to X-Newest in swift). Charge it 10x or something. Sometimes you really need this.
https://lamport.azurewebsites.net/tla/formal-methods-amazon....
In general I'm with you, but that here is the main benefit imho. I don't have the nerves to operate my own redundant, transparently growing and reliable storage system for my business. For use cases where privacy is important (like backups, private user data, etc), encrypting data before uploading limits the privacy leaks to metadata which might be acceptable.
I am genuinely interested in what you think about things like this:
ssh user@rsync.net du -Ahd2 some/directory
pg_dump -U postgres db | ssh user@rsync.net "dd of=db_dump"
ssh user@rsync.net "git clone git://github.com/freebsd/freebsd.git freebsd"
The real argument against S3 though is its price. For any but most trivial amounts of data it's terribly expensive and I use it only as internal storage for other AWS services.
One could self-host minio which is nice S3 replacement: https://github.com/minio/minio
(Assuming an already enterprise-y prosumer IS, i.e. pfSense, VLANs, ZFS storage, 'enough' CPU/RAM/network, all patched and well in good sysadmin fashion, inet-facing resources behind some reverse proxy, etc).
The first target is internal customers for a consulting dev shop — i.e. me + n first partners / founders / employees (I'll begin alone but with a goal to grow to ~3 people asap).
So website, storage w. inet-facing endpoints, security all the way (encrypt etc), and basic IT stuff for dev: e.g. private GitLab + straightforward CI/CD pipe. I figure all developers need is an endpoint to push git, and access to the automation processes.
____
Longer elaboration, FWIW (maybe not much to answer the question but I'm a bit over my head on this topic so I'd rather share too much, to get debunked where needed).
The rationale to bootstrap as self-hosted is simply that it's already paid (gear is humming as I write) and I'm “99% sure I can guarantee 99% uptime” (the full setup is split in two homes 500km apart, each behind 1Gbps symmetrical business-grade fiber, each site being in High-Availability locally— from router/firewalls to storage passing by proxies and k8s-pods).
Re certs, well in a bootstrap scenario we'd worry about those later I suppose, when there's actual demand to justify the overhead cost (first clients are likely to be extremely low-profile, looking for a "break" too). Legally, clients own their code and infra at the end of the day, we would deliver a product/tool but not host it (not an MSP; we'd typically rather work with their MSP— or recommend one —to smooth out their operation, add critical value to key pain points).
In terms of actual workloads, I'm looking at custom domain-parametized ML/DL services (think: selling very specialized neural nets that solve one problem extremely well for this one company, e.g. classifying internal docs). So I need some in-house hardware anyway to experiment when developing models (it's painful and limiting to be forced to spend at the exploratory stage, for each project).
I figure my first partners or employees would either run their stuff locally or tap into my GPUs; we'd likely pay for the big guns on AWS or GC only for actual prod training iterations (those that earn us actual money when our work is 'good enough', final drafts so to speak).
It's a bit of a maverick project¹, hoping to leverage ML in profitable ways for small operations (obviously as we accumulate clients and experience, we'd get better at nailing areas of best-value optimization). It will be a rocky road during the early phase (first two years at least). The costs can skyrocket pretty fast with ML on clouds, I'm extremely wary of such fuckups, not to mention running the ongoing risk of being cancelled or shutdown (account terminated or frozen) for whatever reasons — so better have "plan B"... which historically might have begun as plan A, i.e. self-hosting.
I've also got a long-term angle of building expertise, so I've a positive bias for on-prems: if this thing works and grows enough, we'll definitely look at running the operation in-house, because clouds cost proposition for ML is too steep to scale profitably if it's your core business, or so I hear (and before one says "but Spotify is on whatever cloud", yeah, that's not their core business, they don't sell AI but music; we're the guys who sell AI to Spotify ideally). So, in ML for now, gotta get'em GPUs and TPUs to make a mark. Thus I'd rather learn all of it (myself maybe, as a company surely) sooner rather than later. It's gonna come down to this anyway, if this maverick thing works.
1: "maverick" as of 2020, but I suspect there will be a whole market of such consulting / ML shops by 2030.
However, S3 isn't trying to replace your NAS. S3 offers many use cases where it is the smartest and most viable solution. The most common use case is as centralized block storage for an online web app, that runs on load balancers, a CDN, and distributed servers. If you run an app that is going to run on multiple servers and get load balanced between them, then uploading directly to the servers is complicated. Since a user could upload something to server A one time, and a month later they could request the file while visiting on server B, C, D, E, etc. This centralized S3 platform simplifies the block storage for these large apps.
S3 does offer more advanced features as well that many people aren't educated on. For example, S3 Lifecycle Management allows you to set up timelines so that your file uploads are available on S3 for 30 days (when users are most likely to access them), then automatically move them to reduced redundancy (reduces cost, but at lower 99.9% SLA) when the file is 30-120 days old, and after that it moves to Glacier storage (long term, taped storage, super cheap), for long term storage when it is unlikely to be retrieved but needs to be kept. You can also set up a CDN just for this block storage, with over 200 edge locations all around the world.
S3 also offers HIPAA and PCI compliance. There is also incredibly robust user management (Through AWS IAM). Plus you get 11x9's of availability (99.999999999% availability).
Im not trying to sell you on S3. It definetely isn't for everyone. There are times when I would tell people that it isn't a good fit. But if you work on enterprise apps, it gives you a lot of enterprise level features for month-to-month, (relatively) straightforward pricing, no long term commitements, etc. Which is something that doesn't generally exist in the world of enterprise software or PAAS (which is what S3 really is). So for that, S3 is a breakthrough product.
S3's biggest flaw is its price. If you are using all the enterprise features than it is arguably worth it, especially considering the 11 nines of availability. But for anyone on a non-enterprise app that doesn't need that stuff, there are far cheaper solutions out there for you. Blackblaze B2, Wasabi, Digital Ocean Spaces are three that come to mind.
Seconded on the last two in this list. I use both, for completely personal storage needs. I was lucky enough to get Wasabi's older, cheaper pricing model for the lifetime of my account, but even so it's a hugely attractive option for storing large amounts of data without worrying about local equipment and all the rest of it.
You'd like https://scaleway.com free-tier of 75GB object storage / transfers per month.
Note that those 9s are for durability
From https://aws.amazon.com/s3/faqs/#How_durable_is_Amazon_S3:
"This durability level corresponds to an average annual expected loss of 0.000000001% of objects. For example, if you store 10,000,000 objects with Amazon S3, you can on average expect to incur a loss of a single object once every 10,000 years"
I agree for the most part on cloud, but isn't "another person's computer" just a server? Serverless is more like a service running on a server that requires zero configuration or maintenance for you, just add your code and it works. Typically that means no messing with apache, nginx, no language nor framework setup/configuration, etc...
Whoa, I thought it was: Use a server, use a CDN, shutdown the server and its running serverless. Because everything is getting hitting the CDN and not the actual server.
Obviously it's on a server, but the "serverless" part comes from not having to provision and manage a server. You can put your code a a service and allow those services (S3, Lambda, API Gateway, etc...) to do the "server" work for you.
Serverless is kind of a generic term. I guess hosting sites like GoDaddy that allow you drag and drop some PHP, HTML and JS files into a CPanel type interface are technically serverless, since you're not managing a server. Put some code up there and let their platform run it for you.
For example, going back to a serverless frontend using S3 that I mentioned before. You can host a JavaScript SPA on S3 and use a service like Route 53 for DNS. Your JavaScript can then make API calls to a full-on server that you manage, say an EC2 instance with a LAMP stack or even Azure with C#/.NET. Your front end is serverless because it's just hosted on S3 and you're not maintaining a server, but your backend code would be deployed and served however you like, such as Jenkins and Kubernetes like you mentioned.
AWS lambda is a good intro to the world of serverless computing if you're not familiar with it. While you can write code on your computer and use the AWS CLI to upload it, the AWS console even has a text editor that resembles JS Fiddle. It can receive events from other AWS services and you can write JavaScript, Python, Java, Ruby, C#, Go and PowerShell to process those events and do whatever you like, such as writing a DB record or adding something to an SQS queue. The point is you're writing real production code that does not live on a server you manage.
Forrest Brazeal is an AWS Serverless Hero and enterprise architect who has led cloud adoption initiatives for companies ranging from startups to the Fortune 50.
When I go to the AWS Heroes page, here's how a "Serverless Hero" is defined:
AWS Serverless Heroes are spirited pioneers of the AWS serverless ecosystem. They evangelize AWS serverless technologies online, in person, and via open source contributions to GitHub and the AWS Serverless Application Repository. These Serverless Heroes help evolve the way developers, companies, and the community at large build modern applications.
While the author of the blog doesn't work directly for AWS, it's in his interest to drive more business for them as he consults for businesses bringing more of their needs to AWS.
Not a smooth process.
I believe Wasabi currently charges $5.99 per tb a month which is pretty reasonable imo.
That's my point. You can't actually state the price per TB is N, you can just estimate it and hope it doesn't exceed the estimate. For some companies and scenarios it doesn't matter, for others it's significant enough.
I don't use it, so can't comment on whether it's even the best at the moment, let alone 'all time'.