Hacker News Reader

>... it might make you feel good to pretend that this is really just a variant of storage systems on your local NVMe unit, and that therefore the APIs involved really ought to converge. [...] The author needs to ask themselves: in this cloud technology stack, is there POSIX involved somewhere lower down, where I can't access it? The answer is, of course, "yes". The sort of cloud storage systems described all run on top of POSIX APIs. They provide convenience (cost efficiency is more debatable) compared to the POSIX alternative, but that's because they exist at an entirely different conceptual layer (hence the presence of POSIX anyway, just buried).

Your point about revealing a POSIX that's actually there but hidden and thus visible to low-level Amazon employees building the S3 service and invisible to S3 end customers is true but that isn't the the point of the article. The author is saying there are motivations for a POSIX-like api to also be visible to the end user.

So your explanation of the "hidden POSIX" stack looks like 2 layers: POSIX api <-- AWS S3 built on top of that

Author's essay is actually talking about 3 layers: POSIX <-- AWS S3 <-- POSIX

That's why the blog post has the following links to POSIX-on-top-of-S3-objects :

https://github.com/s3fs-fuse/s3fs-fuse

https://github.com/kahing/goofys

https://www.cuno.io/

Hi, author here.

Yes, you're right—there's POSIX somewhere along the line. Object storage is available on-cloud but it's also frequently used on-prem (e.g. within many companies, universities, research labs, etc). This object storage may run on Linux and make POSIX function calls and syscalls that perform erasure coding onto block storage, but it may not necessarily rely on a POSIX filesystem on the backend (in fact, in most cases, there is no filesystem at all).

> in fact, in most cases, there is no filesystem at all

Huh? With few exceptions, applications write to disk using a VFS provided by an operating system. Can you provide an example of cloud storage that uses raw disks (bypassing VFS)?

There also isn’t a POSIX file system, just POSIX compatible file systems. Those file systems may expose non-POSIX APIs to do other low level work (trim, clone, snapshot, etc.).

CEPH would be one of those, which is more like a database using raw devices, and no linux filesystem (nor VFS calls) in between. It used to run on top of xfs, but now storage nodes (OSDs) talks directly to a raw device (or an lvm on top of it). But you can most certainly not run "ls" on such a host and get a listing of what some user wrote over S3 into it.

Are there any benchmarks that justify bypassing VFS? My Google-fu is failing and the one page I found comparing file systems in the CEPH docs doesn’t talk about raw access, all images are broken (stored on CEPH raw storage?), and the text doesn’t seem to include throughout numbers.

https://www.pdl.cmu.edu/PDL-FTP/Storage/ceph-exp-sosp19.pdf my google-fu found this

it's hard to read and summarize. It seems that the main point is that filesystems don't expose many tools ceph needs

If you store stuff on xfs or ext4, it will cause more writes due to the journal, and since ceph is a DB in some sense, it also keeps a journal/WAL so doubling that is a waste.

Hi, similar to nvm0n2 who already commented below, object storage systems often directly write to block storage rather than through a VFS. While minio, for example, can run on top of a VFS, even they say you shouldn't be - you should instead run it directly on block storage across multiple drives / nodes.

It’s not uncommon for high performance object stores to bypass POSIX FS semantics and provide kernel patches etc - eg lustre do this.

ZFS new object store backend give you that interface I think.

One particular sore spot is the shell in POSIX.2 which Debian thought so highly of that they disposed of bash.

I understand that the POSIX shell must a) compile to a 64k text segment b) with clean code, but is this really the best that we can do?

Perhaps arrays are far too radical, but perhaps we could start with the simple bash-compatibility extensions that Ron Yorston has added to the Almquist Shell in the Windows version of Busybox.

Could the POSIX standard shell move that far? Or is there absolutely no hope?

https://frippery.org/busybox/index.html

> One particular sore spot is the shell in POSIX.2 which Debian thought so highly of that they disposed of bash.

Uh, no, actual reason was that they required old sh compatibility anyway and dash was just much faster (we're talking 2-5x as fast for most scripts) for that

https://lwn.net/Articles/343924/

And it was Ubuntu that started the switch, not Debian, Debian just followed suit. So please don't spew bullshit reasons you think something happened as gospel.

At the time Debian was still using SysV so it was noticeable as sh invocations were far more numerous during boot.

> So please don't spew bullshit reasons you think something happened as gospel.

That's completely uncalled for and obnoxious.

This is very much not true.

"A lot of effort was made to keep ksh88 small. In fact the size you report on Solaris is without stripping the symbol table. The size that I am getting for ksh88i on Solaris is 160K and the size on NetBSD on intel is 135K.

"ksh88 was able to compile on machines that only allowed 64K text. There were many compromises to this approach."

Without a doubt, that had a profound impact upon the POSIX shell. It was also the cause of the removal of many features in the Korn shell for the standard, especially arrays.

This is why POSIX Powershell could never exist.

https://news.slashdot.org/story/01/02/06/2030205/david-korn-...

Interesting how expressed frustration with POSIX.2 is received.

"POSIX file system for the web" is kind of the whole deal for the company that wrote this. They make something to mount object storage as a filesystem.

This isn't a random programmer philosophising about POSIX, it's an ad for a product.

Hi, author here.

Yes, we're a startup. With cunoFS, we've made it possible to leverage the lower cost and higher throughput of object storage like S3, and make it work transparently with functions like e.g. mmap(), chmod(), execve(), renameat2().

> (...) leverage the lower cost and higher throughput of object storage like S3 (...)

What do you mean by "lower cost (...) of object storage like S3"? Isn't S3 terribly expensive? I recall S3 costs around 10$/month per TB you park there without doing anything to it, and AWS charges basically for everything you do to those objects over a network.

Not to mention that apparently we're supposed to compare that to storing on your local harddisk. Isn't local storage always far higher throughput than accessing something across the internet? Being able to store and access stuff on the internet as if it's local is a great idea, but the way they sell it is unconvincing and sounds like they don't really understand what they're talking about.

Hi author here, sorry I missed this post. The performance benchmarks and cost comparisons are for comparing S3 vs EBS (ext4 formatted), EFS, FSx Lustre and others within the same datacenter (i.e. LAN use case rather than WAN use case). That means if you have an EC2 instance running in, say AWS Ohio, and are comparing those storage options also within AWS Ohio, then cunoFS is both cheaper and higher throughput than those other options. It's a different story over WAN. In that case, your own local NVMe storage is going to be cheaper and generally faster that remote storage over a WAN. But that local NVMe storage (on say your solo laptop) isn't going to have anywhere near the Enterprise-grade redundancy, availability and scalability that AWS S3/Azure Blob/Storj/Wasabi/etc has.

It smells like a bunch of FUD to me

Software needs fewer priest wannabes declaring ends of so-called eras. The death cult admiring endless updates and changes needs to be gone

FUD is common both ways. Your comment could be interpreted as FUD against anything new.

I take it you don't use linux, containers, kubernetes, the cloud, or html5 - but lots of other people do ;-)

Yeah, I use all of those. But what I can't really use is some low-rent sales copy trying to persuade me to abandon FOSS tooling in favor of yet another rent-seeking cloud solution.

New != Good

Agreed, with only one slight quibble: content can be good even when the purpose is sales.

I don't know. Consider:

1. We have no idea if S3 uses POSIX to do file IO at the base. By now it wouldn't be surprising if it doesn't, and handles block storage itself bypassing the kernel. The S3 API is more restrictive and so you can probably optimize by not using kernel filesystems. Even Ceph does this now (BlueStore writes directly to flash devices bypassing existing file systems).

2. S3 API and POSIX file IO aren't that different. They certainly aren't dealing with radically different concepts. The differences are more to do with the difficulties of supporting the full featureset in a distributed and performant manner.

S3 is different because it needs/wants to be fully stateless, whereas POSIX is fundamentally stateful (the file descriptor, current directory, etc). It's willing to suffer less efficiency on each procedure call to get that statelessness. And it avoids operations that require strong consistency such as true directories, mtimes, read-after-write, then tosses some stuff that isn't important if you're just storing files like the UNIX DACL permissions.

But ultimately, they're both storing files. The differences are fairly minor.

> 2. S3 API and POSIX file IO aren't that different. They certainly aren't dealing with radically different concepts. The differences are more to do with the difficulties of supporting the full featureset in a distributed and performant manner.

They are hugely different. S3 does not allow you to overwrite data in the middle of an object, while that's a significant use case for POSIX file APIs. S3 does not give you strongly consistent operations, while that's a significant use for POSIX file APIs.

Originally not, but they advertise strong read-after-write consistency as of December 2020.

https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-rea...

As far as I know, you still don't get e.g. create-if-not-exist. The object listing is now strongly consistent, but you don't get to do atomic operations on it.

> By now it wouldn't be surprising if it doesn't, and handles block storage itself bypassing the kernel.

So you imagine a process somehow granted the ability to bypass the kernel and do disk i/o directly from user space. Interesting concept, but I would consider that surprising if true. Bypassing filesystems is entirely likely, but that has nothing to do with bypassing POSIX (e.g. open/read/write/close/ioctl).

Thank you for phrasing this so much more politely than I would have.

I found web development itself to be a pretty frustrating experience, and I am also starting to grow frustrated with web developers. Especially the clueless variety which is nicely on display here.

Makes a guy reach for some impolite words. These days I'm favoring ones that end in "-wad".

It reads to me more like the title is 'flamebait' if you like, and you've been baited by it; what the article actually seems, to me, to be saying is 'POSIX good, but many things use S3-like object storage, we need a POSIX-like standard there, by the way we make cunoFS a POSIX-compat layer for "S3-compatible"'.

Incidentally, my own reading of the title was as if it was going to be defending POSIX against someone who'd said it was outdated. 'POSIX considered outdated' type blog & response.

Except many of those systems are implemented on top of type 1 hypervisors, running either containers or minimal runtimes, and if there is any POSIX related APIs, it is only the minimal surface as required by ISO C and ISO C++ standards.

POSIX as defined is stuck in classical CLI and daemons, it hardly offers APIs for anything else.

fuse is a thing and it suffers from the exact same problem.

You could use FUSE to mount an FTP server as a filesystem but doing a listing on that "directory" may absolutely destroy your performance.

And yet, it's still a very useful thing to do.

The point being, the abstraction isn't going to be perfect, but it doesn't need to be perfect to be useful.

Hell, the extistence of terraform should make that loud and clear.

Yet, the largest issues with networked filesystems is that POSIX forbids canceling file accesses, the lock semantics is really difficult, and cache is hard.

POSIX is absolutely outdated, but this article isn't helpful on understanding why or how.

Anyway, if you create an standard, the cloud providers will make a point of not following it. You can't fix a market by writing something on paper.

I don't quite see how web development is relevant here; I think you've missed the point. Just because it's 'object storage' doesn't mean developers are reaching for React, HTML, CSS, JavaScript, etc. Object storage is a means to put huge volumes of data elsewhere besides your own hard disk/solid state drives, or a NAS running in the basement/attic; that's all. They are a means to access that data at extremely large scale, and can work with native or web apps.

More importantly, object storage need not even exist in the cloud at all; there exist solutions to set up S3 and S3-compatible storage on-premises.

There are plenty of examples of native programs leveraging object storage, whether in the cloud, or on-premises. Consider the latest release of Microsoft Flight Simulator, which streams high-quality assets (city/building 3D models, terrain, textures, weather data, real-time NPC air traffic, etc) in real-time as the user flies over the virtual Earth[1]. The competition has distinctly worse graphical fidelity, primarily because of storage concerns.

Scientific data—especially astrophysics and telescope data—tends to be hosted on blob/object storage (albeit not necessarily on S3) because of the immense data volume involved. These include including Hubble, the Sloan Digital Sky Survey, the Transiting Exoplanet Satellite Survey (TESS), and the Kepler and K2 exoplanet missions.

[1]: https://www.techradar.com/news/microsoft-flight-simulator-20...

Disclaimer: I am an employee at cunoFS.

> There are plenty of examples of native programs leveraging object storage, whether in the cloud, or on-premises. Consider the latest release of Microsoft Flight Simulator, which streams high-quality assets (city/building 3D models, terrain, textures, weather data, real-time NPC air traffic, etc) in real-time as the user flies over the virtual Earth[1]. The competition has distinctly worse graphical fidelity, primarily because of storage concerns

Well that's cool, but why on earth would I expect to use a POSIX API for that, and conversely, why would I expect to use the API that presumably is used for that (decidely non-POSIX-y in most ways) when I want to interact with my own kernel and associated hardware? The tasks are fundamentally different.

"I want object storage APIs for local storage" <= completely reasonable

"I want object storate APIs to be like POSIX or vice versa" <= considerably less reasonable

Why? Because all typical, existing applications written for POSIX systems use the POSIX API (usually indirectly) to interact with the filesystem and perform IO. Being able to use that vast ecosystem of existing applications seamlessly with object storage is such a common requirement that a good dozen different object storage-to-POSIX abstraction layers exist (many of them on top of FUSE).

Even if you build your scalable production environment from scratch with native object storage support (which is rare!), it’s eminently useful to be able to use coreutils to interact with said object storage for devops: nothing beats e.g. running grep on a list of log “files” in an S3 bucket in terms of convenience.

Have you tried running Samba on top of cunoFS ? I'd be interested in performance data on that. For SMB3 access it might be better to add a Samba VFS vfs_cunoFS interface directly. You won't need to open source your client code, just the wrapper.

Hi sorry I missed this. Yes, you can run Samba for SMB, and Ganesha for NFSv4. Feel free to email us if you think there's a better way though.

Yes, I think there's a better way (which would make Samba faster on your system :-). Can you ping me at jra@samba.org and I'll explain !

As a much more low-level developer, I think they have a point, but not that the interface should be exactly S3.

After all, the only thing that's truly "local" is the processor cache, and on the currently executing core at that. Everything else you have to send out for. Even the DRAM is still ~30ns away. The NVMe devices are on the other end of a "network" defined by the PCIe structure.

So many things on the filesystem would prefer blob semantics of all-or-nothing writes. You don't want to get an updated block in the middle of an executable, image, movie, shared library, etc. But there's no way to expose that differing semantics up to applications.

(not everything! sometimes you do actually want a database! But most of the time you don't want to have to do the required POSIX atomic file update dance)

We know that "everything is a file" was never quite true, but it was a useful principle for POSIX. "Everything is a blob" isn't quite true either, but it may be useful.

> The sort of cloud storage systems described all run on top of POSIX APIs

Is that a benefit or a cost?

I'm not disputing the idea that sometimes (much) higher level abstractions in an API are a good idea.

I'm disputing that the POSIX API is designed to serve anything like the same purpose as (say) S3.

If anything web dev is way worse than the POSIX standard as everything changes every 6-12 months.

Hell, why do so many static pages even need javascript

Because obviously we need to generate them from Next.js. /s

It was produced by someone thinking, "why don't I ask AI what is the difference between cloud storage APIs and POSIX".

APIs keep getting conflate for REST APIs. As far as I understand it, the stdlib is an api