Is POSIX really outdated?
cuno.io
cuno.io
The author needs to ask themselves: in this cloud technology stack, is there POSIX involved somewhere lower down, where I can't access it? The answer is, of course, "yes". The sort of cloud storage systems described all run on top of POSIX APIs. They provide convenience (cost efficiency is more debatable) compared to the POSIX alternative, but that's because they exist at an entirely different conceptual layer (hence the presence of POSIX anyway, just buried).
There's about as much reason to expect a single API to access the filesystem attached to the PCI bus and an unstructured storage system as there is to have a single API to create a PDF of a document AND deliver it to the printer. Which is to say: the higher level abstraction will use the lower level anyway, but it exists for a reason, and isn't appropriate for everything.
I found web development itself to be a pretty frustrating experience, and I am also starting to grow frustrated with web developers. Especially the clueless variety which is nicely on display here.
Makes a guy reach for some impolite words. These days I'm favoring ones that end in "-wad".
This isn't a random programmer philosophising about POSIX, it's an ad for a product.
Yes, we're a startup. With cunoFS, we've made it possible to leverage the lower cost and higher throughput of object storage like S3, and make it work transparently with functions like e.g. mmap(), chmod(), execve(), renameat2().
What do you mean by "lower cost (...) of object storage like S3"? Isn't S3 terribly expensive? I recall S3 costs around 10$/month per TB you park there without doing anything to it, and AWS charges basically for everything you do to those objects over a network.
Software needs fewer priest wannabes declaring ends of so-called eras. The death cult admiring endless updates and changes needs to be gone
I take it you don't use linux, containers, kubernetes, the cloud, or html5 - but lots of other people do ;-)
New != Good
Your point about revealing a POSIX that's actually there but hidden and thus visible to low-level Amazon employees building the S3 service and invisible to S3 end customers is true but that isn't the the point of the article. The author is saying there are motivations for a POSIX-like api to also be visible to the end user.
So your explanation of the "hidden POSIX" stack looks like 2 layers: POSIX api <-- AWS S3 built on top of that
Author's essay is actually talking about 3 layers: POSIX <-- AWS S3 <-- POSIX
That's why the blog post has the following links to POSIX-on-top-of-S3-objects :
https://github.com/s3fs-fuse/s3fs-fuse
Yes, you're right—there's POSIX somewhere along the line. Object storage is available on-cloud but it's also frequently used on-prem (e.g. within many companies, universities, research labs, etc). This object storage may run on Linux and make POSIX function calls and syscalls that perform erasure coding onto block storage, but it may not necessarily rely on a POSIX filesystem on the backend (in fact, in most cases, there is no filesystem at all).
Huh? With few exceptions, applications write to disk using a VFS provided by an operating system. Can you provide an example of cloud storage that uses raw disks (bypassing VFS)?
There also isn’t a POSIX file system, just POSIX compatible file systems. Those file systems may expose non-POSIX APIs to do other low level work (trim, clone, snapshot, etc.).
it's hard to read and summarize. It seems that the main point is that filesystems don't expose many tools ceph needs
I understand that the POSIX shell must a) compile to a 64k text segment b) with clean code, but is this really the best that we can do?
Perhaps arrays are far too radical, but perhaps we could start with the simple bash-compatibility extensions that Ron Yorston has added to the Almquist Shell in the Windows version of Busybox.
Could the POSIX standard shell move that far? Or is there absolutely no hope?
Uh, no, actual reason was that they required old sh compatibility anyway and dash was just much faster (we're talking 2-5x as fast for most scripts) for that
https://lwn.net/Articles/343924/
And it was Ubuntu that started the switch, not Debian, Debian just followed suit. So please don't spew bullshit reasons you think something happened as gospel.
At the time Debian was still using SysV so it was noticeable as sh invocations were far more numerous during boot.
"A lot of effort was made to keep ksh88 small. In fact the size you report on Solaris is without stripping the symbol table. The size that I am getting for ksh88i on Solaris is 160K and the size on NetBSD on intel is 135K.
"ksh88 was able to compile on machines that only allowed 64K text. There were many compromises to this approach."
Without a doubt, that had a profound impact upon the POSIX shell. It was also the cause of the removal of many features in the Korn shell for the standard, especially arrays.
This is why POSIX Powershell could never exist.
https://news.slashdot.org/story/01/02/06/2030205/david-korn-...
That's completely uncalled for and obnoxious.
1. We have no idea if S3 uses POSIX to do file IO at the base. By now it wouldn't be surprising if it doesn't, and handles block storage itself bypassing the kernel. The S3 API is more restrictive and so you can probably optimize by not using kernel filesystems. Even Ceph does this now (BlueStore writes directly to flash devices bypassing existing file systems).
2. S3 API and POSIX file IO aren't that different. They certainly aren't dealing with radically different concepts. The differences are more to do with the difficulties of supporting the full featureset in a distributed and performant manner.
S3 is different because it needs/wants to be fully stateless, whereas POSIX is fundamentally stateful (the file descriptor, current directory, etc). It's willing to suffer less efficiency on each procedure call to get that statelessness. And it avoids operations that require strong consistency such as true directories, mtimes, read-after-write, then tosses some stuff that isn't important if you're just storing files like the UNIX DACL permissions.
But ultimately, they're both storing files. The differences are fairly minor.
So you imagine a process somehow granted the ability to bypass the kernel and do disk i/o directly from user space. Interesting concept, but I would consider that surprising if true. Bypassing filesystems is entirely likely, but that has nothing to do with bypassing POSIX (e.g. open/read/write/close/ioctl).
They are hugely different. S3 does not allow you to overwrite data in the middle of an object, while that's a significant use case for POSIX file APIs. S3 does not give you strongly consistent operations, while that's a significant use for POSIX file APIs.
https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-rea...
POSIX as defined is stuck in classical CLI and daemons, it hardly offers APIs for anything else.
After all, the only thing that's truly "local" is the processor cache, and on the currently executing core at that. Everything else you have to send out for. Even the DRAM is still ~30ns away. The NVMe devices are on the other end of a "network" defined by the PCIe structure.
So many things on the filesystem would prefer blob semantics of all-or-nothing writes. You don't want to get an updated block in the middle of an executable, image, movie, shared library, etc. But there's no way to expose that differing semantics up to applications.
(not everything! sometimes you do actually want a database! But most of the time you don't want to have to do the required POSIX atomic file update dance)
We know that "everything is a file" was never quite true, but it was a useful principle for POSIX. "Everything is a blob" isn't quite true either, but it may be useful.
> The sort of cloud storage systems described all run on top of POSIX APIs
Is that a benefit or a cost?
I'm disputing that the POSIX API is designed to serve anything like the same purpose as (say) S3.
More importantly, object storage need not even exist in the cloud at all; there exist solutions to set up S3 and S3-compatible storage on-premises.
There are plenty of examples of native programs leveraging object storage, whether in the cloud, or on-premises. Consider the latest release of Microsoft Flight Simulator, which streams high-quality assets (city/building 3D models, terrain, textures, weather data, real-time NPC air traffic, etc) in real-time as the user flies over the virtual Earth[1]. The competition has distinctly worse graphical fidelity, primarily because of storage concerns.
Scientific data—especially astrophysics and telescope data—tends to be hosted on blob/object storage (albeit not necessarily on S3) because of the immense data volume involved. These include including Hubble, the Sloan Digital Sky Survey, the Transiting Exoplanet Satellite Survey (TESS), and the Kepler and K2 exoplanet missions.
[1]: https://www.techradar.com/news/microsoft-flight-simulator-20...
Disclaimer: I am an employee at cunoFS.
Well that's cool, but why on earth would I expect to use a POSIX API for that, and conversely, why would I expect to use the API that presumably is used for that (decidely non-POSIX-y in most ways) when I want to interact with my own kernel and associated hardware? The tasks are fundamentally different.
"I want object storage APIs for local storage" <= completely reasonable
"I want object storate APIs to be like POSIX or vice versa" <= considerably less reasonable
Even if you build your scalable production environment from scratch with native object storage support (which is rare!), it’s eminently useful to be able to use coreutils to interact with said object storage for devops: nothing beats e.g. running grep on a list of log “files” in an S3 bucket in terms of convenience.
Hell, why do so many static pages even need javascript
Incidentally, my own reading of the title was as if it was going to be defending POSIX against someone who'd said it was outdated. 'POSIX considered outdated' type blog & response.
You could use FUSE to mount an FTP server as a filesystem but doing a listing on that "directory" may absolutely destroy your performance.
And yet, it's still a very useful thing to do.
The point being, the abstraction isn't going to be perfect, but it doesn't need to be perfect to be useful.
Hell, the extistence of terraform should make that loud and clear.
POSIX is absolutely outdated, but this article isn't helpful on understanding why or how.
Anyway, if you create an standard, the cloud providers will make a point of not following it. You can't fix a market by writing something on paper.
Services for UNIX was a different story, but that's not what either version of WSL is.
Who would bother getting POSIX, UNIX, or FIPS 140-2 certified in today's world? We've moved on from when any of those things were important. It costs time/money and what customers would Microsoft attract?
The US Gov't no longer sees those as important requirements.
It meets the same bar that all of the other "mostly compliant" OSes listed on the POSIX wiki page meet: you can write code against a POSIX stdlib and it'll work without involving a VM as long as you stick to some subset of interfaces and behaviors.
WSL1's compliance was actually larger than most of the OSes on that list. The last time I asked Mentor Graphics which parts of POSIX they supported on Nucleus they said "tell us what APIs your application needs and we'll implement them" (anything beyond PSE51 was largely unimplemented). Zephyr is nice enough to actually document their limitations, but the support is far from comprehensive either:
https://docs.zephyrproject.org/latest/services/portability/p...
I don't think that is any better than OSes that conform to some specific subset.
There was also a POSIX file system API available to win32 programs, but I'm not sure how that was related.
POSIX is outdated and problematic. It's outdated because it was conceived when hardware was vastly different than today. Storage used to be orders of magnitude slower than compute. Today, it's the opposite. POSIX APIs make squeezing all the juice out of the underlying hardware impossible. A standard duplex 100Gbps network link will carry ~300M small IP frames per second. You'll be lucky if you can do more than a few 10k/s using POSIX APIs. That's a massive bottleneck.
And it's not something that can be fixed with clever implementations. None of the POSIX APIs are async. So, to drive concurrency, programmers have to resort to threads that don't scale. That's a fundamental issue that no hardware improvement and/or software trickery will ever fix.
Today's reality is that software is seldom written against POSIX but against Linux, which offers many more APIs. Linux is, likewise, not formally standardized. It dodges the problem of S3 because it's open-source and ubiquitous. But that's not a good solution: it stifles innovation. For that reason, there hasn't been any new (production-ready, serious) kernel in decades.
We are at a deadlock, and POSIX is part of the problem.
Historically, buffered IO was sufficient to circumvent slow threads. Simply because buffered IO operations usually don't block (you merely memcpy, and the kernel flushes asynchronously in the background). That can only take you so far, however, and it's apparent today that hardware can go much further and that the gap is widening.
It's a valid point, though, to question whether that additional performance is even needed. John Ousterhout (https://www.youtube.com/watch?v=o2HBHckrdQc) is currently working on a new network protocol to alleviate some of the problems TCP creates in software wrt performance, and he also questions whether there is even a need for very fast networking for real-world applications.
IMO, the mere existence of stuff like DPDK is proof enough. Many folks use it and would rather use the kernel if it could provide comparable performance.
Another classic example is that POSIX does not specify what happens when one program opens a directory stream and starts calling readdir/seekdir and then another program adds entries to the directory. Does the first program get entries in its stream for those members? Who's to say, that's up to the file system implementer.
I don't actually know if S3 guarantees that if a new stream is created with ListBuckets/ListObjects that the stream will return any new bucket/object created before the stream is closed by the client. Are those semantics defined, or is it left up to the implementer too?
That sure is a problem with the file APIs, but far from the only one. Other issues include weird asynchrony, weird scatter/gather, lack of standard control over exposing parallelism to the device, broken fsync() and fdatasync() error semantics, just enough default buffering features to cause confusion but not avoid most performance bugs, lack of exposing device atomic write units, and many other things.
It's an old API we've had for a long time. Made sense 40 years ago. But today it's just too low level to be easy, and too high-level to be fast.
I'd say Vulkan is the best API that follows the Microsoft API design - the initialization process is always "fill out this struct and call vkCreateWhatever", and the struct fields are actually documented well, not "HANDLE handle - this is the handle".
As for COM all over the place, specially after Vista, the idea is great, the execution from multiple C++ libraries, and poor VS tooling, since the OLE 1.0 days not so much.
VB 6 and .NET are the best developer experience for dealing with COM, but WinDev rather impose on us dealing with C++ and clunky tooling.
Linus’s quote in the article isn’t really true. It’s true in that small, incremental changes are often ignored in the interest of portability (I’d rather my app worked on both macos and linux so will use generic facilities when at all possible; the system-specific ones may be easier and better but I’ll just have to write something different for the other system.
But when the change is a big one, the calculus is different. Consider io_uring: it’s a big win so worth writing special code to use. But the trend in these cases is to write a higher level library , or modify an existing library to work with io_uring. Not a lot of people use system calls directly, except perhaps `stat` and a few other filesystem calls like that.
Another case is the procfs, because you can use it directly from the shell.
Inflexible. Poor ACL management. All this recursive bs that I still can't wrap my head around. User groups, not a flat hierarchy? Poor observability.
Windows is terrible in many ways but with files and in comparison to Posix Windows actually got it right.
EDIT: I did not mention the article. It is silly to compare S3 to Posix. However. Azure and other cloud providers that did offer Posix storage have dropped those offerings.
Yes, POSIX-like file access have plenty of rough edges but alternatives are mostly trading this for that, not straight up upgrade. I'd like to see deburred version (cutting all the edge case crap with fsyncing and such, among other problems), but good luck getting everyone on board of that
Every feature comes with drawbacks. For example ability to just lock the file might be great from app perspective but it is PITA from any distributed backend perspective. Ability to change it after write comes with similar problems so S3 and other cloudy ones tend to skip that.
ACLs will always be fucking mess because the more powerful ones you need the more messy access control becomes. And it always becomes married with whatever underlying system uses for authentication.
Like, you can rewrite app to use S3 but suddenly you now can't rewrite header of the file without going full read-modify-write cycle because modifying an object would be hard at scale so Amazon decided to just not do it.
So stream-encoding MP4 file directly to S3 (which at end of encoding requires writing some stuff in header) can't be done directly on S3 and you need a temporary file (guess in what? POSIX local filesystem) before the upload, or use different format.
> Poor observability.
What that has to do with anything ? It's just API, you can observe it as any other API... hell, it's actually far more observable than app using S3 as you can't strace that but you can strace filesystem operations.
Funny you should give the stream-encoding MP4 example, because yes that is what people's experience has been for S3. We've solved that - no temporary local file needed - all streamed directly to S3, for example using ordinary ffmpeg. The trick is a deeper understanding of how multi-part upload works, and if necessary, server-side copy semantics on those parts.
The problems with this approach date back all the way to AFS in the 80s, which was a full-file transfer with a local cache much like an S3 layer has to be. Simple example: it was immediately discovered that nobody ever checks the return code of close(). In a network upload situation you can definitely get an error from close() and you have to deal with it.
Yes agree that you can't "just" put a POSIX API on S3, but that doesn't make it impossible. For the sake of keeping the article to a reasonable size, I left a lot of things out. There are tradeoffs that occur between POSIX semantics, consistency and performance. Each application/process has different needs but the great thing about running right inside the process is that we can see what those needs are and adapt. For example, many applications have no need for random access writes - the only libc calls and syscalls exposed are purely sequential. Some processes have both random access writes and POSIX record locks around them to protect them from other concurrent processes - and we can see that. That means we treat these applications/files differently, with some corresponding performance implications. This is very different to a normal filesystem that has to treat every process the same way because it is a "black box".
You're right that AFS, and for that matter NFS, can in principle return error on close which many existing applications unfortunately aren't written to handle. However, that doesn't mean that NFS isn't practical - it is very widely used.
Our customers mostly run workloads in the same region as the object storage (whether in cloud or on-prem) typically with very high availability. As an essentially networked file system, you're right that it can't make much stronger guarantees than the NFS protocol itself does, but operating inside cloud infrastructure you typically see 4 9s availability.
The strongest argument for using a different API for object storage is that you don’t get that excuse. The API presents the true semantics and failure conditions and the application needs to think them through. (And your position that it may not be a strong enough argument is perfectly valid.)
The BSDs, Mac OS included, have ways to do things that are not in POSIX, and they're usually the best way to get that specific thing done.
Linux is the same, but doesn't follow the BSDs lead on those APIS (epoll, inotify, kqueue/kevent etc).
io_uring is awesome, but there's no analog for FreeBSD etc.
Fuchsia, Windows and other platforms offer Linux compatibility, not as focused on specifically POSIX.
So, is Linux the new POSIX? I don't know...
... but I think about this monthly at least.
https://en.cppreference.com/w/c/23
That list isn't exhaustive.
Cloud-based systems are basically the new UNIX, in that they are all completely different and non-standard and it's impossible to write something portable for more than one cloud provider. It would be great to have a "new POSIX" designed for the Cloud era.
> Object storage is not a filesystem, and was developed with different use cases in mind. Unlike the filesystems POSIX is built to interact with, object storage is unstructured, making it readily scalable, including by pooling storage across different locations
My porn collection[0] is an unstructured object storage, making it scalable because it includes pooled storage across different locations.
[0] Jokes on you, I turned vanilla and offloaded all my requirements to the famous Orange-Black, unstructured, pooled, scalable site. But I have other things what still fit this definition.
> Object storage is usually accessed using RESTful HTTP APIs and is an example of how files can be interacted with in the absence of POSIX.
Somebody, tell the author about SMB and what network access, storage and file storage are all different things.
> At cunoFS, we’ve worked with a number of developers on diverse object-storage-backed solutions for high performance and big data purposes
[Even if didn't notice where this article is hosted] Ah, another pitch from a company deeply interested in you using theirs product?
Yes, we've had to do a lot of things to deal with object storage latency. Since cunoFS is running inside a given process itself, it has greater visibility into what actions it can take and is likely to take. This means we can make huge improvements in our prediction logic, so that we can prefetch within and across files much better, thus hiding latencies. For POSIX metadata, we have a caching mechanism which is shared between processes, and we've invented a better way to encode this metadata so that it is retrieved alongside the LIST operation.
> Why don't more companies adopt ChromeOS as an option. (self.sysadmin)
> submitted 5 days ago by psuedononymoose
> Why spend your time dealing with AD, malware, ransomware, patches etc? I run an org with over 800 adults on ChromeOS and I find myself on an island. I however see a lot of questions and concerns on here that would completely be alleviated by using ChromeOS. What is the hold up? Most everything is a web app now. Honestly trying to understand besides management won't do it
https://old.reddit.com/r/sysadmin/comments/177def3/why_dont_...
Disclaimer: emphasis is mine
Offline mode is a thing, but does not work consistently. There is no conceptual distinction regarding why some apps work offline and some don’t.
Certain things that nearly every computer can do, like say print a file, become impossible.
Every app uses its own unique set of widgets and UI concepts. No two apps, even from the same company work consistently or have the same key bindings and capabilities (Gsuite is mostly consistent, but not with Gmail, and other Google apps, move off of Google and it gets much worse very quickly).
On the other hand, my kids can log in on any device and have their environment (as spartan as that might be) the way they left it. Until they get to older grades they don’t have to learn concepts like folders and files (for better or worse).
For businesses this seems awesome. Call center workers, for example, that get kiosk mode systems that run the CRM, KB, and nothing else seems great, basically like a more flexible POS system, but for every role.
Chromebooks for the call centre will come… may god have mercy on our souls.
It does LD_PRELOAD based dynamic syscall interception - which is dangerous enough, but also doesn't work for a lot of cases.
So they have a JIT ELF translator that will redirect calls for static binaries.
While all this is cool technically, this is not a recipe for stability of your binaries.
I see the point in theory, but it works incredibly well in practice. The specific technology has been used in production by major companies for years. It even carefully works around buggy software that makes incorrect assumptions for undefined behaviour.
(COI disclaimer: I used to work on this product; but I no longer have any stakes in it, financial or otherwise. I do still use the product, because it’s vastly superior to the alternatives.)
I actually did similar in production at Google, which i suspect is one of the companies you are referring to.
I'm even overall a fan of using JITted code in production for normally "static" languages.
But - i've never seen this sort of thing break through the significant resistance/feeling of taboo that often exists around doing that kind of thing in production, long term.
You will find companies here and there willing, sure. But writ large? It eventually goes away, even at companies willing to try it.
I do hope they get past all that, i just ... am skeptical.
Yes we have many large companies (Fortune Global 500) down to small organisations using our software with this kind of interception (see for example https://cuno.io/about-us/). It took us a decent sized team a lot of years to get right, because it is so very hard a problem to crack. But we think it is worth it. And for those who don't want to use such interception, we do offer a FUSE layer as well that still offers much higher performance than alternatives.
Just to clarify: no, I was referring to companies which are using products from the company behind cunoFS, and which share the actual code base of the functional interposition. My point is that while this technique is complex and brittle in general, this specific code-base is incredibly battle-tested and has proved itself even in fairly arcane configurations.
You’re definitely right about there being some amount of resistance, but functional interposition offers some compelling advantages over all alternative solutions in terms of ease of use and unparalleled performance.
but let POSIX die in peace
create new standards, sure
but preferable standards with more narrow scopes, e.g. a remote block storage API standard, which is just that and nothing more
It's almost always a better idea (and not much work) to write your own thin specialized glue layer between your application code and operating system APIs.
The article is about POSIX compliance (or lack thereof) of storage systems.
And the article does not imply that “POSIX … sucks”. On the contrary: the answer to the rhetoric question in the title is obviously “no”.
> It's almost always a better idea … to write your own thin specialized glue layer between your application code and operating system APIs.
Oh, definitely. But writing good abstractions that work equally well with POSIX-compliant filesystems and with object storage is basically impossible without massive trade-offs. That’s the entire point of the product that’s advertised here: it provides a link between object storage and POSIX-compliant file access that manages these trade-offs extremely well, and it allows users to use their existing glue code for POSIX without having to deal with object storage altogether.
(COI disclaimer: I used to work on this product; but I no longer have any stakes in it, financial or otherwise.)
The problem is that it's actually really hard to write you own specialized glue layer,and so most developers who do that have poor implementations with low performance and/or incompatibilities with non-AWS solutions. In the context of object storage, we've found lots of applications have tried to add S3 support, and while they work with AWS S3 and have basic functionality, they fail on a lot of other S3-compatible solutions. So they end up tied to AWS, and you can't use them on Microsoft Azure Storage, or often on Google Cloud Storage (despite its S3 gateway), or others.
For instance, a key workhorse of the genomics field is `samtools`, which works with AWS S3 in some ways, but not others (like Amazon Resource Names[1]). Our approach works across vendors transparently, and on S3, is much faster compared to such native implementations.
[1]: https://docs.aws.amazon.com/IAM/latest/UserGuide/reference-a...