Comment by jasode | Hacker News Reader

Comment by jasode | Hacker News Reader

Parent Full thread

>... it might make you feel good to pretend that this is really just a variant of storage systems on your local NVMe unit, and that therefore the APIs involved really ought to converge. [...] The author needs to ask themselves: in this cloud technology stack, is there POSIX involved somewhere lower down, where I can't access it? The answer is, of course, "yes". The sort of cloud storage systems described all run on top of POSIX APIs. They provide convenience (cost efficiency is more debatable) compared to the POSIX alternative, but that's because they exist at an entirely different conceptual layer (hence the presence of POSIX anyway, just buried).

Your point about revealing a POSIX that's actually there but hidden and thus visible to low-level Amazon employees building the S3 service and invisible to S3 end customers is true but that isn't the the point of the article. The author is saying there are motivations for a POSIX-like api to also be visible to the end user.

So your explanation of the "hidden POSIX" stack looks like 2 layers: POSIX api <-- AWS S3 built on top of that

Author's essay is actually talking about 3 layers: POSIX <-- AWS S3 <-- POSIX

That's why the blog post has the following links to POSIX-on-top-of-S3-objects :

https://github.com/s3fs-fuse/s3fs-fuse

https://github.com/kahing/goofys

https://www.cuno.io/

Hi, author here.

Yes, you're right—there's POSIX somewhere along the line. Object storage is available on-cloud but it's also frequently used on-prem (e.g. within many companies, universities, research labs, etc). This object storage may run on Linux and make POSIX function calls and syscalls that perform erasure coding onto block storage, but it may not necessarily rely on a POSIX filesystem on the backend (in fact, in most cases, there is no filesystem at all).

> in fact, in most cases, there is no filesystem at all

Huh? With few exceptions, applications write to disk using a VFS provided by an operating system. Can you provide an example of cloud storage that uses raw disks (bypassing VFS)?

There also isn’t a POSIX file system, just POSIX compatible file systems. Those file systems may expose non-POSIX APIs to do other low level work (trim, clone, snapshot, etc.).

Hi, similar to nvm0n2 who already commented below, object storage systems often directly write to block storage rather than through a VFS. While minio, for example, can run on top of a VFS, even they say you shouldn't be - you should instead run it directly on block storage across multiple drives / nodes.

CEPH would be one of those, which is more like a database using raw devices, and no linux filesystem (nor VFS calls) in between. It used to run on top of xfs, but now storage nodes (OSDs) talks directly to a raw device (or an lvm on top of it). But you can most certainly not run "ls" on such a host and get a listing of what some user wrote over S3 into it.

Are there any benchmarks that justify bypassing VFS? My Google-fu is failing and the one page I found comparing file systems in the CEPH docs doesn’t talk about raw access, all images are broken (stored on CEPH raw storage?), and the text doesn’t seem to include throughout numbers.

https://www.pdl.cmu.edu/PDL-FTP/Storage/ceph-exp-sosp19.pdf my google-fu found this

it's hard to read and summarize. It seems that the main point is that filesystems don't expose many tools ceph needs

If you store stuff on xfs or ext4, it will cause more writes due to the journal, and since ceph is a DB in some sense, it also keeps a journal/WAL so doubling that is a waste.

It’s not uncommon for high performance object stores to bypass POSIX FS semantics and provide kernel patches etc - eg lustre do this.

One particular sore spot is the shell in POSIX.2 which Debian thought so highly of that they disposed of bash.

I understand that the POSIX shell must a) compile to a 64k text segment b) with clean code, but is this really the best that we can do?

Perhaps arrays are far too radical, but perhaps we could start with the simple bash-compatibility extensions that Ron Yorston has added to the Almquist Shell in the Windows version of Busybox.

Could the POSIX standard shell move that far? Or is there absolutely no hope?

https://frippery.org/busybox/index.html

> One particular sore spot is the shell in POSIX.2 which Debian thought so highly of that they disposed of bash.

Uh, no, actual reason was that they required old sh compatibility anyway and dash was just much faster (we're talking 2-5x as fast for most scripts) for that

https://lwn.net/Articles/343924/

And it was Ubuntu that started the switch, not Debian, Debian just followed suit. So please don't spew bullshit reasons you think something happened as gospel.

At the time Debian was still using SysV so it was noticeable as sh invocations were far more numerous during boot.

This is very much not true.

"A lot of effort was made to keep ksh88 small. In fact the size you report on Solaris is without stripping the symbol table. The size that I am getting for ksh88i on Solaris is 160K and the size on NetBSD on intel is 135K.

"ksh88 was able to compile on machines that only allowed 64K text. There were many compromises to this approach."

Without a doubt, that had a profound impact upon the POSIX shell. It was also the cause of the removal of many features in the Korn shell for the standard, especially arrays.

This is why POSIX Powershell could never exist.

https://news.slashdot.org/story/01/02/06/2030205/david-korn-...

Interesting how expressed frustration with POSIX.2 is received.

> So please don't spew bullshit reasons you think something happened as gospel.

That's completely uncalled for and obnoxious.

ZFS new object store backend give you that interface I think.

Reply on news.ycombinator.com