S3 is files, but not a filesystem
calpaterson.com
calpaterson.com
Believe the hype. S3's durability is industry leading and traditional file systems don't compare. It's not just the software - it's the physical infrastructure and safety culture.
AWS' availability zone isolation is better than the other cloud providers. When I worked at S3, customers would beat us up over pricing compared to GCP blob storage, but the comparison was unfair because Google would store your data in the same building (or maybe different rooms of the same building) - not with the separation AWS did.
The entire organization was unbelievably paranoid about data integrity (checksum all the things) and bigger events like natural disasters. S3 even operates at a scale where we could detect "bitrot" - random bit flips caused by gamma rays hitting a hard drive platter (roughly one per second across trillions of objects iirc). We even measured failure rates by hard drive vendor/vintage to minimize the chance of data loss if a batch of disks went bad.
I wouldn't store critical data anywhere else.
Source: I wrote the S3 placement system.
I'd also assume that a sufficient number of these corruption events are used as a signal to "heal" the system by migrating the individual data blocks onto different machines.
Overall, I'd say the things that you mentioned are pretty typical of a storage system, and are not at all specific to S3 :)
Source: https://docs.aws.amazon.com/AmazonS3/latest/userguide/checki...
I'd rather believe the test results.
Is there a neutral third-party that has validated S3's durability/integrity/consistency? Something as rigorous as Jepsen?
It'd be really neat if someone compared all the S3 compatible cloud storage systems in a really rigorous way. I'm sure we'd discover that there are huge scary problems. Or maybe someone already has?
Not better than all of them.
A geo-redundant rsync.net account exists in two different states (or countries) - for instance, primary in Fremont[1] and secondary in Denver.
"S3 even operates at a scale where we could detect "bitrot""
That is not a function of scale. My personal server running ZFS detects bitrot just fine - and the scale involved is tiny.
[1] he.net headquarters
> S3 even operates at a scale where we could detect "bitrot" - random bit flips caused by gamma rays hitting a hard drive platter (roughly one per second across trillions of objects iirc).
I would expect any cloud provider to be able to detect bitrot these days.
The neat thing about AWS' AZ architecture is that it's a sweet spot in the middle. They're far enough apart for good isolation, which provides durability and availability, but close enough that the network round trip time is negligible compared to the disk seek.
Re: bit rot, I mean the frequency of events. If you've got a few disks, you may see one flip every couple years. They happen frequently enough in S3 that you can have expectations about the arrival rate and alarm when that deviates from expectations.
What may be less of a sweet spot is AWS' pricing.
Back on topic, I'd hope all of us would expect value for money for any and all services we recommend or purchase. Search for "site:news.ycombinator.com Away From AWS" to find dozens of discussions on how to save money by leaving AWS.
EDIT: just one article of the many I've read recently:
"What I’ve always found surprising about egress is just how expensive it is. On AWS, downloading a file from S3 to your computer once costs 4 times more than storing it for an entire month"
https://robaboukhalil.medium.com/youre-paying-too-much-for-e...
Only for spinning rust, right?
It can't possibly be enough for critical data though, right? I'm guessing a fire in 1 is unlikely to spread to another, but could it affect the availability of another? What about a deliberate attack on the DCs or the utilities supplying the DCs?
There have been region-wide availability outages before. They're pretty rare and make worldwide news media due to how much of the internet they take out. I don't think there's been S3 data loss since they got serious about preventing S3 data loss.
Availability is a different beast than durability. I think people are paranoid here about durability instead of availability.
S3 advertises four nines availability and 12 nines durability.
https://docs.aws.amazon.com/AmazonS3/latest/userguide/MultiR...
https://cloud.google.com/storage/docs/locations#consideratio...
I mean, technically it’s not bitrot if zeros were accidentally written out instead of data.
I don’t think this is true. Per the Google Cloud Storage docs, data is replicated across multiple zones, and each zone maps to a different cluster. https://cloud.google.com/compute/docs/regions-zones/zone-vir...
https://cloud.google.com/storage/docs/locations#consideratio...
I only ask because your post is a bit like singing praises for Cinnabon that they make their own dough.
The things that you mentioned are standard storage company activities.
Checksum-all-the-things is a basic feature of a lot of file systems. If you can already set up your home computer to detect bitrot and alert you, you can bet big storage vendors do it.
Keeping track of hard drive failure rates by vendor is normal. Storage companies publicly publish their own reports. The tiny 6-person IT operation I was in had a spreadsheet. Hell, I toured a friend’s friend’s major data center last year and he managed to find time to talk hard drive vendors. Now you. I get it — y’all make spreadsheets.
There are a lot of smart people working on storage outside AWS and long before AWS existed.
As far as I can tell, every cloud provider's object store is too durable to actually measure ("14 9's"), and it's not a problem.
But that's burying the lede. By far the greatest risks to a file's durability are: 1. Bugs (which aren't captured by a durability model). This is mitigated by deploying slowly and having good isolation between regions. 2. An act of God that wipes out a facility.
The point of my comment was that it's not just about checksums. That's table stakes. The main driver of data loss for storage organizations with competent software is safety culture and physical infrastructure.
My experience was that S3's safety culture is outstanding. In terms of physical separation and how "solid" the AZs are, AWS is overbuilt compared to the other players.
I was not at Google for the Clichy fire, but it wasn't the first datacenter fire Google experienced. I think your information about Google's data placement may be incorrect, or you may be mapping AWS concepts onto Google internal infrastructure in the wrong way.
Can we stop this dance on HN? Can you just name them, please?
But on the other hand, GCP supports multi-region so that's not nearly as big of a deal as it would be if AWS zones were not sufficiently isolated
11 9's of durability means mean time to data loss of 100 billion years. Nothing on earth is 11 9's durable in the face of natural (or man-made) disasters. The earth is only 4.5 billion years old.
By the way, I don't doubt that AWS has plenty of 9's by that metric - perhaps more than GCP.
According to https://nuclearsecrecy.com/nukemap/ - it'd take at least a 1 megaton warhead to take out two of the ap-southeast-2 datacenters, and over 10MT to take out 3.
I suspect you'd need a lot less than that though, the 1MT warhead would probably take out enough outside-the-datacenter infrastructure to take the entire AZ offline. I don't care too much though, if someone's dropped a warhead that close to home I have other things to worry about than whether all the cat pictures and audit logs survive.
Speak for yourself. Many of us love our audit logs and show them to strangers whatever we can.
We’re among friends here.
AWS's guarantee is that you can lose two of the three copies and still have all the data. You can't do that without three complete copies.
-1. government, the historical cause of most data loss
1½. google canceling the product and deleting all the data, as they did with google+
How can you tell that if it's not measurable?
As far as I can tell the '11/14 9s' durability numbers are more or less completely made up. That's why AWS doesn't offer any actual durability SLA for S3, only a 99.9% availability SLA[0].
However, Google is very good at not losing data.
"A lot"? Does anything but ZFS and maybe btrfs do this? Ext4 anf XFS — two very common filesystems — still don't have data checksums.
Unfortunately I’m not aware of any filesystem that does it while maintaining the full bandwidth of a modern NVMe. Not even with the extra reads factored in; on ZFS I get 800 MB/s max.
Also: you had to be eighteen or older to operate the mixer. It was something like a 60 quart machine, and all the recipes were pre-programmed for time and power with pauses to change from the paddle to the whip if needed.
If you see this I wonder if S3 is planning on adding hardlinks?
Alternatively, AWS does publicly provide legally binding availability guarantees, but I have never seen any prominently displayed legally binding durability guarantees. Are these published somewhere less prominently?
It's listed prominently in the public docs: https://aws.amazon.com/s3/storage-classes/
Given that their lawyers clearly indicate that "designed for availability" is not what they are contractually obligated to provide, only the letter of the SLA does that; "designed for durability" is similarly a marketing statement that does not incur any contractual obligations. Is there some specific statement in that document that I am missing which indicates that data durability is not fully at their convenience?
I would never ever think of an SLA as anything more than a financial commitment - if you think more of it you'll eventually be in a world of hurt.
The question is: What level of durability does AWS contractually guarantee where a failure to provide that level results in a breach of contract that may incur damages and where, specifically, in the documentation do they specify that?
Outdated anecdata: I've worked for a company that lost some parts of buckets after the lightning strike incident in 2011, which bumped the paranoia quite a bit. AFAIK same thing couldn't happen for more than a decade.
What you say resonates really well with me, and what I've heard during these years.
This misses something critical. Yes, s3 has fast reading and writing, but that’s not really what makes it useful.
What makes it useful is listing. In an unversioned bucket (or one with no delete markers), listing any given prefix is essentially constant time: I can take any given string, in a bucket with 100 billion objects, and say “give me the next 1000 keys alphabetically that come after this random string”.
What’s more, using “/“ as a delimiter is just the default - you can use any character you want and get a set of common prefixes. There are no “directories”, ”directories” are created out of thin air on demand.
This is super powerful, and it’s the thing that lets you partition your data in various ways, using whatever identifiers you need, without worrying about performance.
If listing was just “slow”, couldn’t list on file prefixes and got slower proportional to the number of keys (I.e a traditional unix file system), then it wouldn’t be useful at all.
Therefore they do not get slower proportional to the number of entries and listing based on file prefixes is extremely fast.
You can maybe use “d_off” with readdir in some way, but that’s specific to the filesystem. There’s no portable way to do this with POSIX.
Regardless of if you can do it with a single directory, you can’t do it for all files recursively under a given prefix. You can’t just ignore directories, or say that “for this list request, ‘-‘ is my directory separator”.
The use of b-trees in file systems is completely beside the point.
But as you say, there are filesystem-specific methods or operating-system specific methods to reach the true performance of the filesystem.
It is likely that for maximum performance one would have to write custom directory search functions using directly the Linux syscalls, instead of using the standard libc functions, but I would rather do that instead of paying for S3 or something like it.
The question isn’t if it’s possible, because of course it is, the question is if it’s portable and well supported with the POSIX interface. Because if it’s not, then…
Where did this goalpost come from? S3 is not portable or POSIX compliant.
This functionality does not exist to my knowledge.
ext4 and XFS return directory entries in pseudo-random order (due to hashing), not lexicographically.
For an example, see e.g. https://righteousit.wordpress.com/2022/01/13/xfs-part-6-btre...
If you know a way to return lexicographical order directly from the file system, without the need to sort, please link it.
UNIX (and all operating systems) differentiate between a file and a directory. To list the contents of a directory, you need to make an explicit call. That call might return files or directories.
So to list all files recursively, you need to list, sort, check if an entry is a directory, recurse”. This isn’t great.
I recall there were more edge cases around HTTP headers, but they don't seem to have been recorded as test cases -- it's been too long for me to remember details, I may have simply ran out of time / real world interop got good enough to prioritize something else.
2011 state, search for fails_on_aws: https://github.com/tv42/s3-tests/blob/master/s3tests/functio...
Current state, I can't speak to the exact semantics of the annotations today, they could simply be annotating non-AWS features: https://github.com/ceph/s3-tests/blob/master/s3tests/functio...
I imagine there's at least one POSIX-compatible file system out there that supports another, more performant method of dumping its internal metadata via some system call or another. But then we would no longer be comparing the S3 and POSIX APIs.
Anyway, I guess this is beyond the point of the original commenter above. I would disagree that listing files efficiently is the most useful part of S3. The main value prop is the fact that you can easily upload and download files from a distributed store. Most use cases involve uploading and downloading known files, not efficiently listing millions of files.
[1] https://jvns.ca/blog/2015/03/05/how-the-locate-command-works...
That has always been good enough for me.
Have not used S3, but that is not how I imagined using it.
Say you store uploads associated with a company and a user. You'd maybe naively store them as `[company-uuid]/[user-id].[timestamp]`.
If you need to list a given users (123) uploads after a given date, you'd list keys after `[company-uuid]/123.[date]`. If you need to list all users uploads, you'd list `[company-uuid]/123.`. If you need to get a set of all users who have photos, you'd list `[company-uuid]/` with a Delimiter set to `.`
The point is that it's flexible and with a bit of thought it allows you to "remove all a users uploads between two dates", "remove all a companies uploads" or "remove all a users uploads" with a single call. Or whatever specific stuff is important to your use-case, that might otherwise need a separate DB.
It's not perfect - you can't reverse the listing (i.e you can't get the latest photo for a given user by sorting descending for example), and needs some thought about your key structure.
That some niche edge-case runs efficiently doesn't sound like a defining feature of S3. On the contrary many common operations map terrible to S3, so you kind of need the logic to be elsewhere.
- Listing things is a very common operation to do.
- The POSIX api and the directory/file hierarchy it provides is a restrictive one.
- S3 does not suffer from this, you can recursively list and group keys into directories at “list time”.
- If you find yourself needing to list gigantic numbers of keys in one go, you can do better by only listing a subset. S3 isn’t a filesystem, you shouldn’t need to list 1k+ keys sequentially apart from during maintenance tasks.
- This is actually quite fast, compared to alternatives.
Whether or not you see a use case for this is sort of irrelevant: they exist. it’s what allows you to easily put data into s3 and flexibly group/scan it by specific attributes.
For sure, for maintenance tasks etc. it sounds quite useful. And good hygiene with prefixes sounds like a sane idea. But listing being a critical part of what "makes S3 useful"? That seems like an huge stretch that your points don't seem to address.
Because there is no POSIX api for this. Depending on your requirements and query patterns, you may not need a completely separate database that you need to keep in sync.
Why? If the S3 structure and listing is sufficient, I don't need to store anything else anywhere else.
Many use cases may involve other requirements that S3 can't meet, such as being able to find the same object via different keys, or being able to search through the metadata fields. However, if the requirements match up with S3's structure, then additional services are unnecessary and keeping them in sync with S3 is more hassle than it's worth.
If you think wider, a bucket itself is just a prefix.
But that's not what we are discussing.
This article misunderstood S3 and could as well have the title: "An Airplane is not a Car" :-)
Maybe it's widespread, but I've not encountered it.
Here is the architecture of Amazon Drive and the storage of metadata.
"AWS re:Invent 2014 | (ARC309) Building and Scaling Amazon Cloud Drive to Millions of Users" - https://youtu.be/R2pKtmhyNoA
And you can see the use here at correct time: https://youtu.be/R2pKtmhyNoA?t=546
But in 2020, S3 changed to strong consistency model. There is no need to use DynamoDB now.
"...Finding objects based on other attributes, however, requires doing a linear search using the LIST operation. Because each listing can return at most 1000 keys, it may require many requests before finding the object. Because of these additional requests, implementing attribute-based queries in S3 alone can be challenging..."
But yeah: things like filtering on tags or created at dates requires another approach.
I know what you’re thinking — 2 seconds, that’s faster than I can type the 300 character file key with its pseudo prefixes)!
Ah, but what if you wanted to get 2 files from S3?
Perhaps the use-cases you're talking about are very different from mine. That's possible of course.
But for me, often the slow speed of listing the bucket gets in the way. Your bucket doesn't have to get very big before listing the keys takes longer than reading them. I seem to remember that listing operations ran at sub-1mbps, but admittedly I don't have a big bucket handy right now to test that.
A pathological case would be a prefix with 100 million deleted keys, and 1 actual key at the end. Listing the parent prefix takes a long time in this case - I’ve seen it take several minutes.
If your bucket is pretty “normal” and doesn’t have this, or isn’t versioned, then you can do 4-5 thousand list requests a second, at any given key/prefix, in constant time. Or or you can explicitly list object versions (and not skip deleted keys) also in constant time.
It all depends on your data: if you need to list all objects then yeah it’s gonna be slow because you need to paginate through all the objects. But the point is that you don’t have to do that if you don’t want to, unlike a traditional filesystem with a directory hierarchy.
And this enables parallelisation: why list everything sequentially, when you can group the prefixes by some character (i.e “-“), then process each of those prefixes in parallel.
The world is your oyster.
What did you measure? How did you compare? This claim seems very contrary to my experience and understanding of how things work...
Let me refine the question: did you measure metadata or data operations? What kind of storage medium is used by the filesystem you use? How much memory (and subsequently the filesystem cache) does your system have?
----
The thing is: you should expect, in the best case, something like 5 ms latency on network calls over the Internet in an ideal case. Within the datacenter, maybe you can achieve sub-ms latency, but that's hard. AWS within region but different zones tends to be around 1 ms latency.
This is while NVMe latency, even on consumer products, is 10-20 micro seconds. I.e. we are talking about roughly 100 times faster than anything going through the network can offer.
and in even more detail of different types of EBS/EFS/FSx Lustre here: https://cuno.io/blog/making-the-right-choice-comparing-the-c...
Our AWS spend is high enough to warrant a very close working relationship with AWS so this is something we have worked with you guys on already.
Would you care to elaborate on your experience or use case a bit more? We've made a lot of improvements over the last few years (and are actively working on more), and we have many happy customers. I'd be happy to give a perspective of how well your use case would work with EFS.
Source: PMT turned engineer on EFS, with the team for over 6 years
Services like DataSync show that the underlying infra can be performant. But it feels almost impossible to replicate that on EFS via standard POSIX APIs. And unfortunately one of our use cases depend upon that.
If feels, to me at least, like EFS isn’t where AWSs priorities lie. At least if you compare EFS to FSx Lustre and recent developments to S3. Both of which has been the direction our AWS SAs have pushed us.
This is why there’s a new S3 Express offering that is low latency (but costs more).
Normally, from someone working in the storage, you'd expect tests to be in IOPS, and the goto tool for reproducible tests is FIO. I mean, of course "reproducibility" is a very broad subject, but people are so used to this tool that they develop certain intuition and interpretation for it / its results.
On the other hand, seeing throughput figures is kinda... it tells you very little about how the system performs. Just to give you some reasons: a system can be configured to do compression or deduplication on client / server, and this will significantly impact your throughput, depending on what do you actually measure: the amount of useful information presented to the user or the amount of information transferred. Also throughput at the expense of higher latency may or may not be a good thing... Really, if you ask anyone who ever worked on a storage product about how they could crank up throughput numbers, they'd tell you: "write bigger blocks asynchronously". This is the basic recipe, if that's what you want. Whether this makes a good all around system or not... I'd say, probably not.
Of course, there are many other concerns. Data consistency is a big one, and this is a typical tradeoff when it comes to choosing between object store and a filesystem, since filesystem offers more data consistency guarantees, whereas object store can do certain things faster, while breaking them.
BTW, I don't think most readers would understand Lustre and similar to be the "local filesystem", since it operates over network and network performance will have a significant impact, of course, it will also put it in the same ballpark as other networked systems.
I'd also say that Ceph is kinda missing from this benchmark... Again, if we are talking about filesystem on top of object store, it's the prime example...
I agree that things like dedupe and compression can affect things, so in our large file benchmarks each file is actually random. The small file benchmarks aren't affected by "write bigger blocks" because there's nothing bigger than the file itself. Yes, data consistency can be an issue, and we've had to do all sorts of things to ensure POSIX consistency guarantees beyond what S3 (or compatible) can provide. These come with restrictions (such as on concurrent writes to the same file on multiple nodes), but so does NFS. In practice, we introduced a cunoFS Fusion mode that relies on a traditional high-IOPS filesystem for such workloads and consistency (automatically migrating data to that tier), and high throughput object for other workloads that don't need it.
This is an interesting hack. However, an IOP is an IOP, no matter how good you predicted it and prefetch it so that you hide the latency it's going to be translated to a GetObject.
I think what you really exploited here is that even though S3 is built on HDDs (and have very low IOPS per TiB) their scale is so large that even if you milk 1M+ IOPS out of it AWS still doesn't care and is happy to serve you. But if my back-of-envelope calculation is correct this isn't going to work well if everyone starts to do it.
How do you get around S3's 5.5k GET per second per prefix limit? If I only have ~200 20GiB files can you still get decent IOPS out of it?
and...
> IOPS is a really lazy benchmark that we believe can greatly diverge from most real life workloads
No, it's not. I have a workload training a DL model on time series data which demands 600k 8KiB IOPS per compute instance. None of the thing I tested work well. Had to build a custom one with bare metal NVMe-s.
Our aim is to unleash all the potential that S3/Object has to offer for file system workloads. Yes, the scale of AWS S3 helps, as does erasure coding (which enhances flexibility for better load balancing of reads).
Is it suitable for every possible workload? No, which is why we have a mode called cunoFS Fusion where we let people combine a regular high-performance filesystem for IOPS, and Object for throughput, with data automatically migrated between the two according to workload behaviour. What we find is that most data/workloads need high throughput rather than high IOPS, and this tends to be the bulk of data. So rather than paying for PBs of ultra-high IOPS storage, they only need to pay for TBs of it instead. Your particular workload might well need high IOPS, but a great many workloads do not. We do have organisations doing large scale workloads on time-series (market) data using cunoFS with S3 for performance reasons.
How are you "parallelizing" the ListObjectsV2? The continuation token can be only fed in once the previous ListObjectsV2 response has completed, unless you know the name or structure of keys ahead of time, in which listing objects isn't necessary.
Very effective for our use case.
I think 99% of S3 usage just consists of retrieving objects with known keys. It seems odd to me to consider prefix listing as a key feature.
You can try it yourself: list objects in a bucket prefix with lots of files, and measure the time it takes to list all of them vs. the time it takes to list only a subset of them that share a common prefix.
I'm not sure we agree on the definition of "constant time" here. Just because you get 1000 keys in one network call doesn't imply anything about the complexity of the backend!
Since each request is dependent upon the position received in the last request, 1000 arbitrary keys on your 3rd or 1000th attempt doesn't really help unless you found your needle in the haystack in that request (and in that case the rest of that 1000 key listing was wasted.)
A request to list objects under “foo/“ is a request to list all objects starting with “foo/“, which is constant time irregardless of the number of keys before. Same applies for “foo/bar-“, or any other list request for any given prefix. There are no directories on s3.
dir1/a/000000
dir1/a/...
dir1/a/999999
dir1/b
A flat string-indexed KV store that only supports lexicographic order, without special handling of delimters, needs to traverse 1 million dirents ("a/00000" throuh "a/999999") before arriving at "b".
Thus, simple flat hierarchies are much slower at listing the contents of a single dir: O(all recursive children), vs. O(immediate children) on a "proper" filesystem.
Lexicographic strings cannot model multi-level tree structures with the same complexities; this may give it the reputation of "listing files is slow".
UNLESS you tell the listing algorithm what the delimter character is (e.g. `/`). Then a lexicographical prefix tree can efficiently skip over all subtrees at the next `/`.
Amazon S3 supports that, with the docs explicitly mentioning "skipping over and summarizing the (possibly millions of) keys nested at deeper levels" in the `CommonPrefixes` field: https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-...
I have not tested whether Amazon's implemented actually saves the traversal (or whether it traverses and just returns less results), but I'd hope so.
S3 has no rename or move operation.
Renaming is CopyObject and then DeleteObject.
CopyObject takes linear time to the size of the file(s).
This comes up fairly often when someone has written a lot of files
to the wrong place - moving the files back is very slow.
In a normal file system, renaming a directory is fast O(1), in S3 it's slow O(all recursive children).
And Amazon S3 has not added a delimiter-based function to reduce its complexity, even though that would be easily possible in a lexicographic prefix tree (re-rooting the subtree).
So here the original post has indeed found a case where S3 is much slower than a normal file system.
I was taken aback by this recently. At my coworkers request, I was putting some work into a script we have to manage assets in S3. It has a cache for the file listing, and my coworker who wrote it sent me his pre-populated cache. My initial thought was “this can’t really be necessary” and started poking.
We have ~100,000 root level directories for our individual assets. Each of those have five or six directories with a handful of files. Probably less than a million files total, maybe 3 levels deep at its deepest.
Recursively listing these files takes literally fifteen minutes. I poked and prodded suggestions from stack overflow and ChatGPT at potential ways to speed up the process and got nothing notable. That’s absurdly slow. Why on earth is it so slow?
Why is this something Amazon has not fixed? From the outside really seems like they could slap some B-trees on the individual buckets and call it a day.
If it is a difficult problem, I’m sure it would be for fascinating reasons I’d love to hear about.
I don’t think btrees would help unless you’re doing directory traversals, but even then I suspect that’s not that beneficial as your bottleneck is going to be the network operations and exposed operations. Ultimately, file listing isn’t that critical a use case and typically most use cases are accomplished through things like object lifecycles where you tell S3 what you want done and it does it efficiently at the FS layer for you.
Indeed there’s every reason to believe that a modern file system would perform significantly faster if the hierarchy was implemented as a prefix filter than actually maintaining the hierarchical data structures (at least for most operations). You can guess that this might be the case that file creation is extremely slow on modern file systems (on the order of hundreds or maybe thousands per second on a modern NVME disk that can otherwise do millions of IOPs and listing the contents of an extremely large directory is exceedingly slow)
I think the previous poster could/should have said, "It is not a hierarchical file system and has no concept of directories." where I added the word "hierarchical".
But it's also pretty obvious that was the point.
For example, you could implement POSIX directory semantics on top of S3. About the only POSIX filesystem API you couldn’t implement it append / overwrite (well you could but it might be prohibitively expensive).
But that illusion breaks when you try to do operations you usually do with/on directories.
After reading that, I now understand your comment.
Deleting an S3 bucket is nontrivial!
You can't delete a bucket with objects in it. And you can't just tell S3 to delete all the objects. You need to send individual API requests to S3 to delete each object. Which means sending requests to S3 to list out the objects, 1000 at a time. Which takes time. And those list calls cost money to execute.
This is a good summary of the situation: https://cloudcasts.io/article/deleting-an-s3-bucket-costs-mo...
The fastest way to quickly dispose of an S3 bucket turns out to be to delete the AWS account it belongs to.
The article has a mistake about this too: expirations do NOT count as lifecycle transitions and you don't get charged as such. You will, of course, get charged if you prematurely delete objects that are in a storage class with a minimum storage duration that they haven't reached yet. This is what they're actually talking about when they mention Infrequent Access and other lower tiers.
Stop being silly, as if 1 vs 2 API calls matters. You should empty large buckets with lifecycle policies. It's trivial.
Imagine you are using windows for the first time and you want to delete a directory, so you find an answer on Serverfault that explains that to do so you need to spin up a COM object that marks the directory for deletion, then the next day comes back and deletes it.
You might be inclined to say ‘that seems overly complicated’.
The original answerer is confused though. ‘It’s trivial, stop being silly. Can you think of a simpler way to delete a directory?’
Do you see now why I thought the ‘non triviality’ of deleting an S3 bucket was perhaps relevant in a discussion on an article about why S3 is both simpler and more complex than a file system?
And why your approach might not actually be making the case for it being as simple as you think?
This is only needed if you have a huge (100 million+) bucket, at which point you should be experienced with s3, otherwise you can just click the big, clear and obvious “empty bucket” button on the console.
It's common to store metadata on DynamoDB where it can be queried, and just have whatever arbitrary links to the values in the buckets.
There's no "recursive" nature to S3 buckets. "Listing a directory" is simply listing keys by a prefix.
So list by the upper-most prefix that you want. If you have 1,000,000 files, it will take 1,000 API calls to list everything.
If each call takes 1s (I have no idea what your latency to the S3 bucket region is), then it will indeed take 15 min.
https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObje...
They fixed it already, it's called DynamoDB. With some SQS and Lambda glue you can index your S3 content in any way you want for later retrieval.
Alternative theory: everyone who worked on this knew that it was not a filesystem and "object store" is a description intended to describe everything else pointed out in this post.
"Objects were really popular" is about objects as software component that combines executable code with local state. None of the original S3 examples were about "hey you can serialize live objects to this store and then deserialize them into another live process!" It was all like "hey you know how you have all those static assets for your website..." "Objects" was used in this sense in databases at the time in the phrase "binary large object" or "blob". S3 was like "hey, stuff that doesn't fit in your database, you know...objects...this is a store for them."
This is meant to describe precisely things like "listing is slow" because when S3 was designed, the launch usecases assumed an index of contents existed _somewhere else_, because, yeah, it's not a filesystem. it's an object store.
To quote GCP:
> Object storage is a data storage architecture for storing unstructured data, which sections data into units—objects—and stores them in a structurally flat data environment
> https://cloud.google.com/learn/what-is-object-storage
That is (1) unstructured (2) flat organization (3) whole-item operations (read, write)
The thing I would expect from a file abstraction is mutability. I should be able to edit pieces of a file, grow it, shrink it, read and write at random offsets. I shouldn't have to go back up to the root, or a higher level concept once I have the file in hand. S3 provides a mutable listing of immutable objects, if I want to do any of the mutability business, I need to make a copy and re-upload. As originally conceived, the file abstraction finds some sectors on disk, and presents them to the client as a contiguous buffer. S3 solves a different problem.
Many people misinterpret the Good Idea from UNIX "everything is a file" to mean that everything should look like a contiguous virtual buffer. That's not what the real Good Idea is. Really: everything can be listed in a directory, including directories. There will be base leaves, which could be files, or any object the system wants to present to a process, and there will be recursive trees (which are directories). The directories are what make the filesystem, not the type of a particular leaf. Adding a new type of leaf, like a socket or a frame buffer, or whatever, is almost boring, and doesn't erode the integrity of the real good idea. Adding a different kind of container like a list, would make the structure of the filesystem more complex, and that would erode the conceptual integrity.
S3 doesn't do any of these things, and that's fine. I just want a place to put things that won't fit in the database, and know they won't bitrot when I'm not looking. The desire to make S3 look more like a filesystem comes from client misunderstanding of what it's good at/for, and poor product management indulging that misunderstanding instead of guarding the system from it.
I don't agree with defining a filesystem as something that has to be backed by a block device, but the shape of a filesystem API is historically very different from the shape of the S3 API.
I agree. To me the correct analog for S3 is a block storage device (a very weird one where blocks can be any size and can have a key associated with them) and not a filesystem. A filesystem is an abstraction that sits on top of a block storage device and so an "S3 filesystem" would have to be an abstraction that sits on top of S3 as the underlying block storage.
That is a very weird one
A block device itself is an abstraction built on real hardware. "Write these 300 bytes" really means something like "move needle on platter 2 to position 6... etc"
S3 is just a different abstraction that is also built on raw storage somehow. It's a strictly flat key-object store. That's it. I don't know why people have a problem with this. If you need "filesystem stuff" then implement it in your app, or use a filesystem. You only need to append? Use a database to keep track of the chain of appends and store the chunks in S3. Doesn't work for you? Use something else. Need to "copy"? Make a new reference to the same object in your db. Doesn't work for you? Use something else.
S3 works for a lot of people. Stop trying to make it something else.
And stop trying to change the meaning of super well-established names in your field. A filesystem is described in text books everywhere. S3 is not a filesystem and never claimed to be one.
Oh and please study a bit of operating system design. Just a little bit. It really helps and is great fun too.
Briefly, Apache OpenDAL is a library providing FS-like APIs over multiple storage backends, including S3 and many other cloud storage.
A few database systems, such as GreptimeDB and Databend, use OpenDAL as a better S3 SDK to access data on cloud storage.
Other solutions exist to manage filesystem-like interfaces over S3, including Alluxio and JuiceFS. Unlike Apache OpenDAL, Alluxio and JuiceFS need to be deployed standalone and have a dedicated internal metadata service.
Here are some related codes on how we implement such a layer in GreptimeDB:
* https://github.com/GreptimeTeam/greptimedb/blob/v0.7.0/src/o... * https://github.com/GreptimeTeam/greptimedb/blob/v0.7.0/src/m...
I’m pretty sure the only other large object storage provider that is v4 only is Azure, and even then they offer a compatibility layer. Backblaze just flat out won’t work unless you pay extra to connect to them.
Honestly the only cloud provider I think you’re talking about is Azure, I don’t know of any other that are IPv4 only because it’s just cost prohibitive.
they does not seem to understand users on the b2 product. it's almost as if b2 is just a supplementary service from their backup service.
https://www.reddit.com/r/backblaze/comments/ij9y9s/b2_s3_not...
https://old.reddit.com/r/backblaze/comments/1av4r3g/b2_ipv6_...
* their weekly 2 hour maintenance window 11:30-13:30 PST (which usually has no downtime, but sometimes is a full outage in the middle of the US day)
* having to file support tickets when your error rates increase above a usable threshold (for us about once a year for the last few years)
* support which does not look into the issue, just asks tons of questions as if they do not have error logs or any visibility on their end
* false success on uploads where B2 says it successfully saved your file but it is 0 bytes on their system (ALWAYS verify the upload despite B2's success code)
* extended outages if there's a high severity CVE (ex: they shut down for 10 hours for the Log4j2 CVE)
They have the best price - but when comparing options, it is simply not a directly comparable product to more mature cloud storage services.
(edit: formatting)
After a lot of iterating we eventually came up with the VFS layer in rclone which adapts S3 (or any other similar storage system like Google Cloud Storage, Azure Blob, Openstack Swift, Oracle Object Storage, etc) into a POSIX-ish file system layer in rclone. The actual rclone mount code is quite a thin layer on top of this.
The VFS layer has various levels of compatibility "off" where it just does directory caching. In this mode, like the article states you can't read and write to a file simultaneously and you can't write to the middle of a file and you can only write files sequentially. Surprisingly quite a lot of things work OK with these limitations. The next level up is "writes" - this supports nearly all the POSIX features that applications want like being able to read and write to the same file at the same time, write to the middle of the file, etc. The cost for that though is a local copy of the file which is uploaded asynchronously when it is closed.
Here are some docs for the VFS caching modes - these mirror the limitations in the article nicely!
https://rclone.org/commands/rclone_mount/#vfs-file-caching
By default S3 doesn't have real directories either. This means you can't have a directory with no files in, and directories don't have valid metadata (like modification time). You can create zero length files ending in / which are known as directory markers and a lot of tools (including rclone) support these. Not being able to have empty directories isn't too much of a problem normally as the VFS layer fakes them and most apps then write something into their empty directories pretty quickly.
So it is really quite a lot of work trying to convert something which looks like S3 into something which looks like a POSIX file system. There is a whole lot of smoke and mirrors behind the scene when things like renaming an open file happens and other nasty corner cases like that.
Rclone's lower level move/sync/copy commands don't bother though and use the S3 API pretty much as-is.
If I could change one thing about S3's API I would like an option to read the metadata with the listings. Rclone stores modification times of files as metadata on the object and there isn't a bulk way of reading these, you have to HEAD the object. Or alternatively a way of setting the Last-Modified on an object when you upload it would do too.
I wonder if you couldn't hack this in by storing the metadata in the key name itself? Obviously with the key length limit of 1024 you would be limited in how much metadata you could store, but it's still quite a lot of space, even taking into account the file path. You could use a deliminator that would be invalid in a normalized path, like '//', for example: /path/to/file.txt//mtime=1710066090
You would still be able to fetch "directories" via prefixes and direct files by using '<filename>//' as the prefix.
This kind of formatting would probably make it pretty incompatible with other software though.
Agree. In MinIO (disclaimer: I work there) we added a "secret" parameter (metadata=true) to include metadata and tags in listings if the user has the appropriate permissions. Of course it being an extension it is not really something that you can reliably use. But rclone can of course always try it and use it if available :)
> You can create zero length files ending in /
Yeah. Though you could also consider "shared prefixes" in listings as directories by itself. That of course makes directories "stateless" and unable to exist if there are no objects in there - which has pros and cons.
> Or alternatively a way of setting the Last-Modified on an object when you upload it would do too.
Yes, that gives severe limitations to clients. However it does make the "server" time the reference. But we have to deal with the same limitation for client side replication/mirroring.
My personal biggest complaint is that there isn't a `HeadObjectVersions` that returns version information for a single object. `ListObjectVersions` is always going to be a "cluster-wide" operation, since you cannot know if the given prefix is actually a prefix or an object key. AWS recently added "GetObjectAttributes" - but it doesn't add version information, which would have fit in nicely there.
Is this "secret" parameter documented somewhere? Sounds very useful :-) Rclone knows when it is talking to Minio so we could easily wedge that in.
> My personal biggest complaint is that there isn't a `HeadObjectVersions` that returns version information for a single object. `ListObjectVersions` is always going to be a "cluster-wide" operation, since you cannot know if the given prefix is actually a prefix or an object key
Yes that is annoying having to do a List just to figure out which object Version is being referred to. (Rclone has this problem when using --s3-list-version).
Simple doesn't mean "not deep". It means having the fewest parts needed in order to accomplish your requirements.
If you require a distributed, centralized, replicated, high-availability, high-durability, high-bandwidth, low-latency, strongly-consistent, synchronous, scalable object store with HTTP REST API, you can't get much simpler than S3. Lots of features have been added to AWS S3 over the years, but the basic operation has remained the same.
That is exactly what "deep" means, in the terminology of this post (from Ousterhout's book A Philosophy of Software Design). Simple means "not complex" (see also Rich Hickey's talk Simple Made Easy: https://www.infoq.com/presentations/Simple-Made-Easy/), while "deep" means providing/having a lot of internally-complex functionality via a small interface. The latter is a better description of S3 (which is what you seem to be saying too) than "simple" which would mean there isn't much to it.
- Easy is when something does not require much effort.
- Simple means the least complex it can be and still work.
- Complex means there are lots of components.
- Easy is a concept that distinguishes the amount of work needed to use a solution
- Simple is a concept that distinguishes whether or not there is an excess number of interacting properties in a system
- Complex is a concept describing the quality of having a number of interacting properties in a system
"Deep", in the context of software complexity, probably only makes sense in terms of describing the number of layers involved in a piece of technology. You could make something have many layers, and it could still be simple, or be complex, or easy.
Files and folders are used to make S3 buckets more approachable to those who either don't know or don't want to know what it actually is, and one day they get a surprise.
Does anyone actually think this? I have never encountered anyone who has described S3 in these terms.
> The concept of deep vs shallow modules comes from John Ousterhout's excellent book. The book is [effectively] a list of ideas on software design. Some are real hits with me, others not, but well worth reading overall. Praise for making it succinct.
It is a great concept and also a great book. I really enjoyed it but I've never found a convincing way to persuade people to read it. I read it on personal recommendation but that only works if it comes from someone you respect (as in my case).
So what's the current best (preferably statically linked) self-hosted, single-node option for minimal S3 like "thing" that just lets me CRUD the files and list them?
But they deprecated it.
(You can still use it, but it's not getting updates.)
Hudi, Delta, iceberg bridge that gap now. Databricks built a company around it.
Don't try to do relational on object storage on your own. Use one of those libraries. It seems simple but it's not. Late arriving data, deletes, updates, primary key column values changing, etc.
Why would we need to do it on object storage which addresses a different type of storage need.
Nevertheless there are projects like EMRFS and S3 file system mount points that try to provide files stem interfaces to workloads that need to see S3 as a filesystem.
*can be used
*file system
(Apologies for typos. The "noprocrast" setting sometimes locks us out of HN right after submitting a comment. And it is now too late, not editable)
It has become a de-facto standard for distributed, data-intensive workloads like those common with spark.
A key benefit is decoupling the data from the compute so that they can scale independently. EBS is tightly coupled to iops and you pay extra for that.
(Source: a long time working in data engineering)
Experienced Spark / Data Engineering teams would not assume S3 is readily useable as a filesystem.
This [1] seems like a good guide on how to configure spark for working with Cloud object stores, while recognizing the limitations and pitfalls.
[1]: https://spark.apache.org/docs/latest/cloud-integration.html
---
Amazon EMR offers a managed way to run hadoop or spark clusters and it implements an "EMR FS" [2] system to interface with S3 as storage.
[2]: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-fs.h...
AWS Glue is another option which is "serverless" ETL. Source and Destination can be S3 data lakes read through a data catalog (hive or glue data catalog). During processing AWs Glue can optionally use S3 [3,4,5] for shuffle partition.
[3]: https://aws.amazon.com/blogs/big-data/introducing-amazon-s3-...
[4]: https://docs.aws.amazon.com/glue/latest/dg/monitor-spark-shu...
[5]: https://aws.amazon.com/blogs/big-data/introducing-the-cloud-...
The problem that emrfs is trying to solve doesn't cover the rdbms scenarios like row-level updates and deletes.
I suppose that's not so different from a WMP user's epiphany when they discover processes, shells, etc.
A document editor or text editor opens files and saves files, but these are whole-document operations. I can’t open a document in Sublime Text without reading it, and I can’t save part of a file without saving all of it. So it’s not obvious that these would be different at an OS level.
As the post points out, there are uses for Unix’s sub-file-level read-and-write commands, but I’ve never needed them.
Like literally any abstraction out there, filesystems are associated with a multitude of possible approaches with conceptually different semantics. It's a bit sophistic to say that Postgres cannot be run on S3 because S3 is not a filesystem; a better choice would have been to explore the underlying assumptions; (I suspect latency would kill the hypothetical use case of Postgres over S3 even if S3 had incorporated the necessary API semantics - could somebody more knowledgeable chime in?).
A more interesting venue to pursue would be - what other additions could be made to the S3 API to make it more usable on its own right - for example, why doesn't S3 offer more than one filename per blob? (e.g., a similar to what links do in POSIX)
There's also the OG Aurora whitepaper: https://www.amazon.science/publications/amazon-aurora-design...
[0] https://people.eecs.berkeley.edu/~matei/papers/2020/vldb_del...
That would not be a great base to build a transactional database on.
Whether something is or isn't a filesystem requires defining what that actually is. A system that stores files would be a simple explanation. Which is clearly something S3 is capable of. This probably upsets the definition gatekeepers for whatever more specific definitions they are guarding. But it has a nice simple logic to it.
It's worth considering that file systems have had a long history, weren't always the way they are now, and predate the invention of relational databases (like postgres). Technically before hard disks were invented in the fifties, we had no file systems. Just tapes and punch cards. A tape would consist a single blob of bits, which you'd load in memory. Or it would have multiple such blobs at known offsets. I had cassettes full of games for my commodore 64. But no disk drive. These blobs were called files but there was no file system. Sometime, after the invention of disks file systems were invented in the early sixties.
Hierarchical databases were common before relational databases and filesystems with directories are basically a hierarchical database. S3 lacking hierarchy as a simpler key value store clearly isn't a hierarchical database. But of course it's easy to mimic one simply by using / characters in the keys. Which is how the fuse driver probably fakes directories. And S3 even has APIs to listfiles with a common prefix. A bigger deal is the inability to modify files. You can only replace them with other files (delete and add). That kind of is a show stopper for a database. Replacing the entire database on every write isn't very practical.
* https://www.amazon.science/publications/amazon-aurora-design... * https://d1.awsstatic.com/events/reinvent/2019/REPEAT_Amazon_...
I am wondering if s3fs/rclone-mount is sufficient, or if I should use something like JuiceFS that adds random-access, renaming, etc on top of it. Are those really necessary APIs for my use case? Is there only one way to find out?
(The app doesn't have native S3 support)
I think a better question is whether there are any good filesystem implementations on top of S3. There are many attempts like s3fs-fuse[^1] or seaweedfs[^2], but I have not heard many stories about their use at scale from big companies. Just recently there was a post here about cunoFS[^3]. It is a startup that implements a POSIX-compliant (supports symlinks, hard links (emulated), UIDs & GIDs, permissions, random writes, etc.) filesystem on top of S3/AZ/GCP storage and claims to have really good performance. I think only time will tell if it works out in practice for companies to use S3 as a filesystem through fs implementations on top of S3.
[^1]: https://github.com/s3fs-fuse/s3fs-fuse
To say you can’t build an efficient database on top of S3 makes sense to me. S3 is already a certain kind of data-storing abstraction optimized for certain usages. If you try and build another data-storing abstraction optimized for incompatible usages on top of that, you are going to have a difficult time.
The issue with POSIX filesystems is that they are required to make a set of tradeoffs to support features a database engine doesn't need, to the significant detriment of scalability and performance in areas that databases care about a lot. For example, one such database filesystem I've used occasionally over the years, while a bit dated at this point, is designed such that you can have tens of millions of files in a single directory where you are creating and destroying tens of thousands of files every second, on upwards of a petabyte of storage. Very far from being POSIX compatible but you don't get anything like that type of scalability on POSIX.
Object storage is far from ideal as database storage. The biggest issue, though, is the terrible storage bandwidth available in the cloud. It is a small fraction of what is available in a normal server and modern database engines are capable of fully exploiting a large JBOD of NVMe.
Oracle DB for a long time supported running on raw partitions which I think suggests that the answer is "not really". Snowflake (and I hear Clickhouse) can run on S3 which I think is more evidence against running on a filesystem. Not to mention the torrid time Postgres has had with fsync on linux.
You can read the research paper here if you are curious: https://www.hopsworks.ai/research-papers/hopsfs-s3-extending...
The reason I don't think about flies on a UNIX-derived systems as a key-value store (filename->file content), is that in such systems, we have many things that aren't really files that expose a file system interface regardless.
AWS is S3, google is buckets, Azure is blob storage, the open source version is … ?
As to your question, object storage[1] seems to be the generic term for the technology. Internally they all rely on naming files based on the hash of their contents for quick lookup, deduplication, and avoiding name clashes.
If you stand up an S3 instance with Ceph, you still have a filesystem on spinning rust or fancy SSDs. There’s just a bunch of stuff on top of that. It’s cool, but to say that there’s no filesystem is simply what the customer or middle person sees, not what is actually happening.
[1]: https://aws.amazon.com/blogs/storage/how-automated-reasoning...
Filesystems are not free; they incur "complexity" (that favorite bugbear everyone on HN loves to complain about) just as much as any other component in the stack does.
> If you stand up an S3 instance with Ceph,
Okay, but AWS does not run on Ceph. Even then, Ceph is an example that recommends the opposite. Nowadays they recommend solutions like the Bluestore OSD backend to store actual data directly on raw block devices, completely bypassing the filesystem layer -- for the exact same reasons I outlined above and many, many others (the actual metadata does use "BlueFS" which is a small FS shim, but this is mostly so that RocksDB can write directly to the block device too, next to the data segments, and BlueFS is in no way a real POSIX filesystem, it's just a shim for existing software).
See "File Systems Unfit as Distributed Storage Backends: Lessons from 10 Years of Ceph Evolution" written by the Ceph authors[1] about why they finally gave in and wrote Bluestore. The spoiler alert is they got rid of the filesystem precisely because "a filesystem with files" underneath, as you describe, was problematic and worked poorly in comparison (see the conclusion in Section 9.)
Many places do use POSIX filesystems for various reasons, even at large scale, of course.
https://ceph.com/community/new-luminous-bluestore/
[Disclaimer: ex-Ceph employee, from before BlueStore]
It would be a poor engineering choice to build something like S3 on top of some other filesystem. There are often ways to do it by using an overlay that converts a filesystem into a pseudo block device, but that is usually considered a compatibility shim used for environments that lacking dedicated storage, at the cost of robustness and performance.
type IBackend interface {
Ls(path string) ([]os.FileInfo, error)
Cat(path string) (io.ReadCloser, error)
Mkdir(path string) error
Rm(path string) error
Mv(from string, to string) error
Save(path string, file io.Reader) error
Touch(path string) error
}
Like yes, for your mum's use case, with a single user, it's probably not all that important that you cover those edge cases, but every time I've built pseudo-filesystems on top of non-filesystem storage APIs, those sorts of semantic questions have been where all the problems have hidden. It's not particularly hard to implement the interface you've described, but it's very hard to do it in such a way that, for example, you never have dangling files that exist but aren't contained in any folder, or that you never have multiple files with the same path, and so on.
https://en.wikipedia.org/w/index.php?title=Comparison_of_fil...
When people decide that X is a filesystem, they try to use it like it's a local, POSIX filesystem, and that's terrible because it won't be immediately obvious why it's a stupid plan.
Can we really blame AWS?
I am sure none of official AWS documentations or examples show such an architecture.
----
Amazon EMR can run Hadoop and use Amazon S3 as storage via EMR FS.
"S3 mountpoints" are a feature specifically for workloads that need to see S3 as a file system.
For block storage workloads there is EBS and EFS and FSx that AWS heavily advertises.
(Apologies for typos. The "noprocrast" setting sometimes locks us out of HN right after submitting a comment. And it is now too late, not editable)
Maybe S3 isn't a filesystem according to this definition, but does it really matter to make it one? I doubt it. The Elastic Filesystem is also an AWS product, but you can't really work as one as you have locally, any folder over 20k files basically will timeout if you do a ls. Does it make EFS a filesystem or not?
https://en.wikipedia.org/wiki/Andrew_File_System was interesting, I'd actually love to see something similar re-implemented with modern ideas, but it's more of an direct-access archival system than a general-purpose filesystem[1], you can't just put files written by arbitrary software on it. It's a bit like NFS without locks&leases, but even less like a normal filesystem; only really good for files created once that "settle down" into effectively being read-only.
[1]: I wrote https://github.com/bazil/plop that is (unfortunately undocumented) content-addressed immutable file storage over object storage, used in conjunction with a git repo with symlinks to it to manage the "naming layer". See https://bazil.org/doc/ for background, plop is basically a simplification of the ideas to get to working code easier. Site hasn't been updated in almost a decade, wow. It's in everyday use though!
there are so many applications depends on file storage, such as Mysql. But horizontal scale for those app still difficult in many case. Replace from vfs api to s3 storage perhaps is trending in my experience.
Except they can be. You don't need to overwrite the whole DB file on every INSERT/UPDATE/DELETE; those can be (and often are) stored in memory and periodically checkpointed. You might lose some writes if the process goes down between checkpoints, but for a lot of applications that's entirely acceptable.
Indeed, for SQLite in particular there are tools like Litestream that support replication to and restoration from S3.
Alternately, you could split the DB across multiple files, and then an INSERT/UPDATE/DELETE would only need to overwrite the files actually affected. This is already how server-based RDBMSs usually work.
server: cloudflare
You said it though - the reason is that its static without any js/frameworks/SPA round trip requests.
But there are still ways to be quicker. For example, the header photo is smaller than the vector diagrams on the page, by about tenfold.
I’ve written my own FUSE that uses Rabin Chunking and stores the data (and meta) in S3. The C++/AWS SDK FUSE is connected to a Go SMB server that runs locally on my Mac and works with (local) TimeMachine.
I use Wasabi for cost and speed reasons.
You can browse, search and sort the files and directories of the different snapshot or versions of the file.
I love it !
For me it's a file system in S3.
Bonus: you must use a key, to encrypt the files.
This seems mistaken. Porting databases that run on local disk to S3 seems like a good way to get a lashing from https://aphyr.com/
Can any databases do it correctly?
If so, I doubt they work with the model of partial overwrites. They probably have to do something very custom, and either sacrifice a lot of tail latency, or their uptime is capped by the uptime of a single AWS availability zone. Doesn't seem like a great design.
(copy of lobste.rs comment)
Directly exposing every write to S3 gives you the partial overwrite issues as described. But one can collect a bunch of traffic and push state to S3 once it reaches a threshold. Instead, a few writes in the postgres WAL are held outside of S3 in a replicated on-disk cache.
But I searched the docs for "durability" and got zero results. Before I use anything like this, I'd like to see what durability settings are used:
https://www.postgresql.org/docs/current/non-durability.html
Litestream documents the their data loss window, it seems like Neon should too:
By default, Litestream will replicate new changes to an S3 replica every second. During this time where data has not yet been replicated, a catastrophic crash on your server will result in the loss of data in that time window.
I also searched for "data loss" and got zero results -- this is important because Neon is almost certainly sacrificing durability for performance.
The link I gave above explains it, right the sentence with "durability":
> Safekeepers are responsible for durability of recent updates. Postgres streams Write-Ahead Log (WAL) to the Safekeepers, and the Safekeepers store the WAL durably until it has been processed by the Pageservers and uploaded to cloud storage.
> Safekeepers can be thought of as an ultra reliable write buffer that holds the latest data until it is processed and uploaded to cloud storage. Safekeepers implement the Paxos protocol for reliability.
If there's no claim, then it's impossible to test :)
In particular, there are no numbers in the description you quoted.
Litestream gives a relatively weak claim, but it could be tested, which actually gives me more confidence in it.
If you look at what aphyr writes, a lot of it is claims from vendors that turned out to be false - https://aphyr.com/tags/jepsen
Try to imagine your astonishment if a traditional storage vendor showed up and told you that their very expensive premium file system they had just sold you:
- can't store log files because it can't append
anything to an existing files
- can't copy files more than 5GB
- can't rename or move a file
Also, is S3 really “very expensive”? Relative to what?
being a conventional filesystem for S3 would be either a very leaky abstraction or completely different product
You can rename / move a file, but it involves copying and deleting the original; I don't understand why they don't have a shortcut for that, but it probably makes sense that the user of the service is aware of the process instead of hiding it.
I'm not sure about the 5GB limit, it's probably documented somewhere as to why that is; possibly, like tweets, having an upper limit helps them optimize things. Anyway there too there's tools, you can do multipart somethings and there's this official blogpost on the subject: https://aws.amazon.com/blogs/storage/copying-objects-greater...
Interesting to note maybe in the context of the post; copy, rename, moving large files, all that could be abstracted away, but that would hide the underlying logic - which might lead to inefficient usage of the service - and worse, make users think it's just a filesystem and use it accordingly, but it's not intended or designed for that use case.
Some people are creating tools that make those services easier to synch with file systems but that is not intended use anyway.
And that's also why you can't append. If you had multiple readers while appending, and appending to multiple replicas, guaranteeing that each reader would see a consistent only-forwards read of the append is extremely hard. So simply ban people from doing that and force them to use a different system designed for the purpose of logging.
Microservices. S3 is for blobs. If you want something that isn't a blob, use a different microservice.
It is also a fiction! There are no folders in S3.
> When you create a folder in Amazon S3, S3 creates a 0-byte object with a key that's set to the folder name that you provided. For example, if you create a folder named photos in your bucket, the Amazon S3 console creates a 0-byte object with the key photos/. The console creates this object to support the idea of folders.
https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-...
Imagine you have a file named /some/dir/file.jpg.
In a filesystem, there’s an inode for /some. It contains an entry for /some/dir, which is also an inode, and then in the very deepest level, there is an inode for /some/dir/file.jpg. You can rename /some to /something_else if you want. Think of it kind of like a table:
+-------+--------+----------+-------+
| inode | parent | name | data |
+-------+--------+----------+-------+
| 1 | (null) | some | (dir) |
| 2 | 1 | dir | (dir) |
| 3 | 2 | file.jpg | jpeg |
+-------+--------+----------+-------+
+-------------------+------+
| key | data |
+-------------------+------+
| some/dir/file.jpg | jpeg |
+-------------------+------+
You can, if you want, use a completely different character to separate “components” in S3, rather than using /, because / is not special. If you want something like “some:dir:file.jpg” or “some.dir.file.jpg” you can do that. Again, because / is not special.
> the API implies that common prefixes indicate related objects.
That’s something users do. The API doesn’t imply anything is related.
And prefixes can be anything, not just directories. If you have /some/dir/file.jpg, then you can query using /some/dir/ as a prefix (like a directory!) or you can query using /so as a prefix, or /some/dir/fil as a prefix. It’s just a string. It only looks like a directory when you, the user, decide to interpret the / in the file key as a directory separator. You could just as easily use any other character.
If you have something which is dir/file, then NORMALLY “dir” does not exist at all. Only dir/file exists. There is nothing to rename.
If you happen to have something which is named “dir”, then it’s just another file (a.k.a. object). In that scenario, you have two files (objects) named “dir” and “dir/file”. Weird, but nothing stopping you from doing that. You can also have another object named “dir///../file” or something, although that can be inconvenient, for various reasons.
More like O(max(number of files, total file size)). You can’t rename objects in S3. To simulate a rename, you have to copy an object and then delete the old one.
Unlike renames in typical file systems, that isn’t atomic (there will be a time period in which both the old and the new object exist), and it becomes slower the larger the file.
Querying ids by prefix doesn’t make any sense for a normal ID type. Just making this operation available and part of your public API indicates that prefixes are semantically relevant to your API’s ID type.
I can look up names with the prefix “B” and get Bart, Bella, Brooke, Blake, etc. That doesn’t imply that there’s some kind of semantics associated with prefixes. It’s just a feature of your system that you may find useful. The fact that these names have a common prefix, “B”, is not a particularly interesting thing to me. Just like if I had a list of files, 1.jpg, 10.jpg, 100.jpg, it’s probably not significant that they’re being returned sequentially (because I probably want 2.jpg after 1.jpg).
Fair enough, basing folders on object names split by / is pretty inefficient. I wonder why they didn't go with a solution like git's trees.
What, exactly, is inefficient about it?
Think for a moment about the data structures you would use to represent a directory structure in a filesystem, and the data structures you would use to represent a key/value store.
With a filesystem, if you split a string /some/dir/file.jpg into three parts, “some”, “dir”, “file.jpg”, then you are actually making a decision about the tree structure. And here’s a question—is that a balanced tree you got there? Maybe it’s completely unbalanced! That’s actually inefficient.
Let’s suppose, instead, you treat the key as a plain string and stick it in a tree. You have a lot of freedom now, in how you balance the tree, since you are not forced to stick nodes in the tree at every / character.
It’s just a different efficiency tradeoff. Certain operations are now much less efficient (like “rename a directory” which, on S3, is actually “copy a zillion objects). Some operations are more efficient, like “store a file” or “retrieve a file”.
This does not make S3 a bad storage, just a bad filesystem, not everything needs to be a filesystem.
Arguably is it good that S3 is not a filesystem, as it can be a leaky abstraction eg in git you cannot have two tags name "v2" and "v2/feature-1" as you cannot have both a file and a folder with the same name.
For something more closely related to URLs than filenames forcing a filesystem abstraction is a limitation as "/some/url", "/some/url/", and "/some/url/some-default-name-decided-by-the-webserver" can be different.[1]
[0] where a different tradeoff is that searching a file by name is slower but reading many small files can be faster.
[1] maybe they should be the same, but enforcing it is a bad idea
Alas, no. It represents a tag, e.g. «folder/», that points to a zero byte object.
You can then upload two files, e.g. «folder/file1.txt» and «folder/file2.txt», delete the «folder/», being a tag, and still have the «folder/file1.txt» and «folder/file2.txt» file intact in the S3 bucket.
Deleting «folder/» in a traditional file system, on the other hand, will also delete «file1.txt» and «file2.txt» in it.
However, there are file managers, FTP clients, and S3 clients that will do that for you by deleting individual files.
So essentially an S3 file manager must be able to handle the situation where there are files without a "directory"—and that I assume is also the most common case as well for S3. Might just not have the "directories" in the first place.
They appear to exist because the key is split on the slash character for navigation in the web front-end. This gives the familiar appearance of a filesystem, but the implementation is at a much higher level.
S3 is not a filesystem because the semantics are different from the kind of semantics we expect from filesystems. You can’t take the high-level API provided by a filesystem, use S3 as the backing storage, and expect to get good performance out of it unless you use a ton of translation.
Stuff like NFS or CIFS are filesystems. They behave like filesystems, in practice. You can rename files. You can modify files. You can create directories.
For example, maybe you have a database mapping file paths to S3 objects. This gives you a separate metadata layer, with S3 as the storage layer for large blocks of data.
It's an incorrect take that a system to manage files must follow a set of patterns like the ones you mentioned to be called "file system".
I would argue that not supporting folders or many other file operations make something not a filesystem today.
I could create a system not supporting folders because it relies on tags or something else. Or I could create a system which is write-only and doesn't support rename or delete.
These systems would be file systems according to how the term has been used for 40 (?) years at least. Just don't see any point in restricting the term to exclude random variants.
Many tools end up handling this by showing a folder named "a" containing a folder named "" (empty string). This confuses users quite a bit. It's more than the inodes, it's how the tooling handles the abstraction.
This created a prefix named “/“ which confused the hell out of the application.
Not only you cannot rename a single file, but you also cannot rename a "folder" (because that would imply a bulk rename on a large number of children of that "folder")
This is the fundamental difference between a first class folder and just a convention on prefixes of full path names.
If you don't allow renames, it doesn't really make sense to have each "folder" store the list of the children.
You can instead have a giant ordered map (some kind of b-tree) that allows you for efficient lookup and scanning neighbouring nodes.
The console UI shows folders but they don’t actually exist in S3. They’re made up by the UI.
You can’t examine when a folder was created (it doesn’t exist in the first place), you can’t rename a folder (it doesn’t exist), you can’t delete a folder (again, it doesn’t exist).
If you truly believe S3 has absolutely no connection to folders, you would answer Yes and No.
“Any hierarchical path structure is a folder” is maybe your definition of “folder”, from what I can tell. I would say that S3 lets you treat paths as hierarchical, but that S3 does not have folders—obviously I have a different definition of “folder” than you do.
We’ve discovered that we have different definitions of “folder”, and therefore, we are not going to agree about whether it is true that “S3 does not have folders” unless we have an argument about what the correct definition of “folder” is. I’m not really interested in that discussion—it’s enough to understand what somebody means when they say “S3 does not have folders” even if you think their definitions are wrong.
Folders are an important part of the way most people use filesystems.
Of course it can still be useful to group objects in the S3 UI, but it would probably be better to use some kind of prefix-centric UI rather than reusing the folder metaphor when it doesn't match the paradigm people are used to.
S3 doesn't have folders. The UI fakes them by creating a 0-byte object (or file, if you will). It's a kludge.
On the Mac, the Finder lets you have files with slashes in their names, even though it's a Unix file system underneath. Don't believe me? Go try to use the Finder to make a directory whose name is "Reports from 2024/03/10". See?
But as everyone knows, slash is the ONLY character you're not allowed to have in a file or directory name under Unix. It's enforced in the kernel at the system call inteface. There is absolutely no way to make a file with a slash in it. Yet there it is!
The original MacOS operating system used the ":" character to delimit directory names, instead of "/", so you could have files and directories with slashes in their names, justs not with colons in their names.
When Apple transitioned from MacOS to Unix, they did not want to freak out their users by reaming all their files.
So now try to use the Finder (or any app that uses the standard file dialog) to make a folder or file with a ":" in its name on a modern Mac. You still can't!
So now go into the shell and list out the parent directory containing the directory you made with a slash in its name. It's actually called "Reports from 2024:03:10"!
The Mac Finder and system file dialog user interfaces actually switche "/" and ":" when they show paths on the screen!
Try making a file in the shell with colons in it, then look at it in the finder to see the slashes.
However, back in the days of the old MacOS that permitted slashes in file names, there was a handy network gateway box called the "Gatorbox" that was a Localtalk-to-Ethernet AFP/NFS bridge, which took a subtly different approach.
https://en.wikipedia.org/wiki/GatorBox
It took advantage of the fact (or rather it triggered the bug) that the Unix NFS implementation boldly made an end-run around the kernel's safe system call interface that disallowed slashes in file names. So any NFS client could actually trick Unix into putting slashes into file names via the NFS protocol!
It appeared to work just fine, but then down the line the Unix "restore" command would totally shit itself! Of course "dump" worked just fine, never raising an error that it was writing corrupted dumps that you would not be able to read back in your time of need, so you'd only learn that you'd been screwed by the bug and lost all your files months or years later!
So not only does NFS stand for "No File Security", it also stands for "Nasty Forbidden Slashes"!
https://news.ycombinator.com/item?id=31820504
>NFS originally stood for "No File Security".
>The NFS protocol wasn't just stateless, but also securityless!
>Stewart, remember the open secret that almost everybody at Sun knew about, in which you could tftp a host's /etc/exports (because tftp was set up by default in a way that left it wide open to anyone from anywhere reading files in /etc) to learn the name of all the servers a host allowed to mount its file system, and then in a root shell simply go "hostname foo ; mount remote:/dir /mnt ; hostname `hostname`" to temporarily change the CLIENT's hostname to the name of a host that the SERVER allowed to mount the directory, then mount it (claiming to be an allowed client), then switch it back?
>That's right, the server didn't bother checking the client's IP address against the host name it claimed to be in the NFS mountd request. That's right: the protocol itself let the client tell the server what its host name was, and the server implementation didn't check that against the client's ip address. Nice professional protocol design and implementation, huh?
>Yes, that actually worked, because the NFS protocol laughably trusted the CLIENT to identify its host name for security purposes. That level of "trust" was built into the original NFS protocol and implementation from day one, by the geniuses at Sun who originally designed it. The network is the computer is insecure, indeed.
[...]
From the Unix-Haters Handbook:
https://archive.org/stream/TheUnixHatersHandbook/ugh_djvu.tx...
Don't Touch That Slash!
UFS allows any character in a filename except for the slash (/) and the ASCII NUL character. (Some versions of Unix allow ASCII characters with the high-bit, bit 8, set. Others don't.)
This feature is great — especially in versions of Unix based on Berkeley's Fast File System, which allows filenames longer than 14 characters. It means that you are free to construct informative, easy-to-understand filenames like these:
1992 Sales Report
Personnel File: Verne, Jules
rt005mfkbgkw0 . cp
Unfortunately, the rest of Unix isn't as tolerant. Of the filenames shown above, only rt005mfkbgkw0.cp will work with the majority of Unix utilities (which generally can't tolerate spaces in filenames).
However, don't fret: Unix will let you construct filenames that have control characters or graphics symbols in them. (Some versions will even let you build files that have no name at all.) This can be a great security feature — especially if you have control keys on your keyboard that other people don't have on theirs. That's right: you can literally create files with names that other people can't access. It sort of makes up for the lack of serious security access controls in the rest of Unix.
Recall that Unix does place one hard-and-fast restriction on filenames: they may never, ever contain the magic slash character (/), since the Unix kernel uses the slash to denote subdirectories. To enforce this requirement, the Unix kernel simply will never let you create a filename that has a slash in it. (However, you can have a filename with the 0200 bit set, which does list on some versions of Unix as a slash character.)
Never? Well, hardly ever.
Date: Mon, 8 Jan 90 18:41:57 PST
From: sun!wrs!yuba!steve@decwrl.dec.com (Steve Sekiguchi)
Subject: Info-Mac Digest V8 #3 5
I've got a rather difficult problem here. We've got a Gator Box run-
ning the NFS/AFP conversion. We use this to hook up Macs and
Suns. With the Sun as a AppleShare File server. All of this works
great!
Now here is the problem, Macs are allowed to create files on the Sun/
Unix fileserver with a "/" in the filename. This is great until you try
to restore one of these files from your "dump" tapes, "restore" core
dumps when it runs into a file with a "/" in the filename. As far as I
can tell the "dump" tape is fine.
Does anyone have a suggestion for getting the files off the backup
tape?
Thanks in Advance,
Steven Sekiguchi Wind River Systems
sun!wrs!steve, steve@wrs.com Emeryville CA, 94608
https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObje...
I’m fine with it, I actually appreciate the logic and simplicity behind it, but the amount of times I’ve tried to explain why “folders” on S3 keep disappearing while people stare at me like I’m an idiot is really frustrating.
(When you remove the last file in a “folder” on S3, the “folder” disappears, because that pattern no longer appears in the bucket k/v dictionary so there’s no reason to show it as it never existed in the first place).
I use S3 just as a web bucket of files (I know it's not the best way to do that but it's what I could easily obtain through our company's processes). But in this case it makes a lot of sense though I try to avoid making folders. But other people using the same hosting do use them.
(also don't get me started on the whole s3api thing)