Hacker News Reader

Comment by drej | Hacker News Reader

A lot of people mistake S3 for a filesystem. S3 and other object stores have very different characteristics, both in terms of performance (latency, troughput) and access patterns (seeks, atomicity). All abstractions on top of objects stores have been really painful for me at work, because people would just ignore the differences between a local filesystem and an object store and were wondering why their perf was terrible.

View on HN

Not only that, by using this you will increase your S3 bill, by definition of how this works.

If you are going to use windows explorer and enter a directory full of images — windows explorer will fetch thumbnail for _every_ image. If you have 50000 files there, that's 50000 requests + bandwidth for 50000 files.

Same for video or audio. Getting metadata isn't free.

But the worst case scenario if you have a big big file, let's say vhdx disk image, that you mount and keep modifying every minute.

S3 protocol doesn't have 'modify', it will _have_ to reupload entire file every time even if a single byte is changed. For a 10G file changed 1000 times that's 10 terabytes of traffic.

Also executable prefetching in windows.

I'm increasingly of the opinion that it would be great to see operating systems provide filesystem APIs with blob storage semantics alongside the regular APIs, and extend this to storage devices. After all, even on the local machine, latency is a big deal and immutability potentially valuable.

Filesystems are allowed to be pretty weird (afs, proc, sshfs, nfs, etc). But yep user expectations have to be calibrated.

I imagine lots of constructs either silently fail, or have complex incomplete substitutes with side effects. Empty directories, symlinks, buffering, Unix domain sockets, named pipes, hard links, sticky bits, dup(), O_TRUNC, O_APPEND, O_EXCL, flock(), creat(), allowed filename characters, metadata like permissions or attributes, and so on.

How do object stores like Ceph get away with having VM's run on them?

Ceph's S3-compatible object store is a layer on top of its core distributed storage. There's also a block storage layer, which is used to run VMs. :-)

Reply on news.ycombinator.com