The original goal of the POSIX standard was application source code portability. However, modern applications are no longer being written to standardized POSIX interfaces.
Speak for yourself.
Speak for yourself.
Actually the inverse: don't speak for yourself, because it's irrelevant.
What the authors did was try to speak for what happens at large -- which is what matters. Outliers will always exist, but their team analyzed thousands of packages across three different OS environments to come to conclusions.
It's not about there being counter-examples, it's about what they saying being the norm.
The notable change between now and 25 years ago is that the POSIX layer tends to be used at the lower levels of system frameworks, while the programs that run on those OSes tend to be written on top of higher level frameworks and do not directly use the POSIX APIs.
This is fine and makes sense given how much has changed in the last 25 years, but given that the POSIX layer is no longer being used (statistically speaking) for what it was originally designed for, it raises questions about to what degree that design is still valid, and whether or not it should be revisited to better suit what it's actually being used for.
I don't think anyone has any answers, but it's always good to question assumptions and never let anything in tech become a sacred cow. Maybe POSIX is fine for its current role, but that doesn't mean people shouldn't think about it.
Disclaimer: I skimmed the article pretty aggressively, so I may have misunderstood something.
POSIX compliance is much more common in consumer OSes nowadays, to the point that every major desktop and mobile platform is built on a POSIX-compliant kernel and runtime except for Microsoft's offerings. In that sense POSIX has taken over the non-Microsoft world, but now it's being used as a much lower-level compatibility layer for system-level frameworks rather than what it was originally designed for.
Even Windows has had multiple POSIX implementations over the years, they were just built as compatibility layers on top of the native APIs. So POSIX-compliant operating systems are almost universal at this point, but POSIX-compliant applications/daemons/etc. (all the programs that run on those POSIX OSes) are rarer than ever.
Yes. Part of this is because more operating systems are approaching POSIX-compliance; but the number of patches we need to port random Linux code to run on FreeBSD has dropped tremendously over the past 20 years.
Obviously not, since you can't go and compile it on another POSIX system -- you are "abstraction compliant" (or rather "abstraction dependent" now.
> There is some sort of perverse pleasure in knowing that it's basically impossible to send a piece of hate mail through the Internet without its being touched by a gay program. That's kind of funny.
Or is this one of those pre-emptive "called it first!" kind of comments?
I didn't spend all that much time, so I'm sure I missed a lot, but I recommend taking a look for yourself as it is quite interesting.
https://en.wikipedia.org/wiki/Apache_Portable_Runtime
That's one of the competitors to POSIX for application portability. Unlike POSIX, it was totally open as well vs people telling me The Open Group will sell me a copy of full standard.
No.
https://news.netcraft.com/archives/category/web-server-surve...
I should add that this is for performance critical software.
On an out-of-the-box installation of FreeBSD/TrueOS, most processes listed by "ps" show a wchan of things like ttyin, select, poll, pause, uwait, wait, or nanslp. On the FreeBSD machine that I am typing at right now exactly one program is waiting in kqread, one of the inner processes of Chrome.
Install my nosh toolset, and a fair amount of that changes, because my programs use kqueue+kevent and suddenly the "ps" output is full of kqreads. But what's left is still not "hardly anyone", by a long chalk.
Would that it were! There's a nasty kernel panic somewhere in the innards of the kevent system on FreeBSD, where a sleeping thread holds a non-sleepable lock, that only manifests itself (at least for me) when there are a lot of programs using kqueue+kevent on a system that is doing a lot of stuff. If more people used kqueue+kevent in more programs, there would be more people wanting it fixed. (-:
For example, the MIO library in Rust, is an abstraction over epoll, kqueue, and the MS primative. Not one over the POSIX standard. This means that if you use that in a Rust library, you get that platform independence for free.
The place you need kqueue and epoll is when most of your connections are idle -- IRC servers and web servers with large keepalive times. But the vast majority of processes are not IRC or web servers.
See http://man7.org/linux/man-pages/man3/basename.3.html for the manpage.
Most computer users access countless sites which run on deep software stacks running on any flavour of linux.
...which run POSIX.
More accurate to say, expose a POSIX compliant API.
I remember BeOS had a 100% compliant POSIX layer. Impressive for a non-UNIX.
This is their application list:
https://github.com/columbia/libtrack/blob/master/workloads/o...
Which are anyway not POSIX.
https://www.amazon.de/MAC-OS-Internals-Amit-Singh/dp/0321278...
That's hardly encouraging, when all the basics and all the newer OS X abstractions are based things other than POSIX standards, and ask programmers to connect to them with non-POSIX interfaces.
The fact that it also supports a POSIX layer is about as relevant to what we're discussing (whether POSIX is fit for today's needs and used in today's mainstream development) as the presence of some X Windows translation layer is in Wayland.
UNIX got X Windows in 1987.
Sure it is down there on the OS tooling, but how many people outside UNIX devs, do actually bother using it?
Personally it doesn't offer me any benefit over an higher level programming language REPL.
There's a reason GUI starts with a G. Other than "graphical" user interfaces, "other" user interfaces will also always have their place..
In some ways looking at how much non-developer computer usage in the past decades has been in front of excel/access sheets and tables, these aren't so "graphical" either. They essentially end up evolving into (haphazard) custom CLI-like keyboard-driven power tools, half-programmable, half requiring intimate familiarity with its "commands" (formulas) or even "training" to use them fluidly.
But if you'd rather code out your CLI needs in a PL REPL (aka CLI), more power to you ;D
Find all files containing a particular string in a directory tree. For each of those files, if its filename appears in a file called Whitelist, write it to a file called Output. Otherwise, do nothing.
Here is one possible command line version (I haven't tested it as I'm on mobile):
TMPF=`mktemp /tmp/XXXXXXXX` && ag MyString | awk -F: '{print $1}' | sort | uniq > $TMPF && comm -12 Whitelist $TMPF > Output && rm $TMPF
comm -12 Whitelist <(ag MyString | awk -F: '{print $1}' | sort | uniq) > Output
comm -12 Whitelist <(ag MyString | awk -F: '{print $1}' | sort -u) > Output
comm -12 Whitelist <(ag -l MyString | sort) > OutputAnd you presumably have to clutter up your filesystem with random project files, etc, which is annoying. Are you going to have 1000 of these, one for every time you need to do some sort of minor task like this?
That command, on the other hand, took me 2 or 3 minutes to write, and I had to look up the syntax for `mktemp`. If I had remembered the mktemp syntax it would have taken a few seconds.
import glob
import shutil
whitelist = set(open('Whitelist').read().split())
for fd in [ x for x in glob.glob('mydir/*', recursive=True) if x in whitelist]:
shutil.copy(fd, 'OUTPUT')
Windows NT was designed towards those promises, but eventually Microsoft realised their mistake and created - ten years ago - Powershell.
The entire dev world has been catching on to the fact that you get reproducible procedures when you use a CLI. Data scientists are another group who know how much power they get from a CLI.
Yet the myth of the obsolete CLI and the omnipotent GUI still is strong. Especially with those people who never really worked with a decent CLI.
And it has, except for backend programmers. Most Windows programmers stay all day in Visual Studio and the like, for example.
But as far as end users as concerned, that is the 99% of computer users, the CLI might as well not exist.
The CLI offers an infinitely combinable and customizable development experience, where huge ammounts of innovation have been occuring. You can see this in modern web development toolchains and new build tools for new languages, which simply would not happen if you restrict your Dev experience to just the GUI. I keep going back to just creating Makefiles for simple automation tasks.
Even the GUIs are starting to learn from this, which is why I think VSCode and Atom are becoming popular; they make it very simple to integrate new CLI tools into the GUIs workflow.
No, I'm speaking from the perspective of someone that has worked in the old days with Sun OS (pre-Solaris) workstations and HP-UX machines, has used (for years) actual VT102 terminals, started using Linux distros around '97 and has been using the CLI in various forms since the mid-eighties or so.
There's nothing limited about the GUI (in theory, and, for most things that matter, in practice too). For example a visual flow language (think Automator in OS X or Quartz Composer, etc) can achieve all the "configurable pipeline" stuff people like in the CLI in a more controlled and formal way.
I also dislike the "dynamic typing" (everything is text/bytes) in the CLI, and would prefer something like PowerShell to have been the norm (with the accompanying toolset). Also note that there's nothing about the GUI that prevents someone from typing some commands too -- and in fact that's part of how lots of GUIs work.
Also, I wouldn't point to "modern web development toolchains" as something to advertise CLIs -- what would that be, Gulp, Grunt, Webpack and the like? All crufty solutions to inadequacies in steering the underlying language properly.
Even in theory, CLIs are not inherently more powerful, it's the other way around (they lack a dimension that GUIs offer (graphics) -- GUIs on the other hand don't lack any dimension -- they can incorporate text commands and text fields and pipelines just fine.
Anybody who has experience from 2-5 different Windows shops and been to a few relevant conferences can make a quite accurate assessment of what "most Windows programmers" do.
That they don't miss it either, and that the "endless, tedious clickfests" are a myth of people who don't know the other side.
Far from this being the reversal that you propound, actual history is even more in favour of your argument. (-:
What does that even mean?
I think you're painting an incorrect picture of the design goals of NT. The original NT was designed by a group of people (e.g. Cutler) with VMS background and they sure as hell didn't have any GUI in mind. In fact, Windows came to the picture much later. Whether NT is administered by a GUI or CLI is totally irrelevant to NT design. Today all the NT derivatives (Windows Server) can be administered with a rich variety of command line tools.
Stdout exists because the original subsystems (OS/2, Windows, MS-DOS) required that. Same applies TODAY. Where is the NT bias to GUIs exactly?
I mean there may not be as many users, but they are massively more powerful (in the realm of data processing only, alas) than people who don't know the CLI. It may be obvious to us, but I should perhaps also point out that CLI ~= REPL for the language that you're using (Bash or whatever).
Sure, these days CLI may mean "a Jupyter notebook", but the same principle applies.
It may not always look the same, but the point is that a CLI/REPL should be a combinatorial force multiplier. Otherwise, it's not really worthwhile.
From what I can tell POSIX has been replaced by per language stdlibs, with the portability of your software tied to the portability of a languages stdlib.
POSIX was mainly designed for C, most other languages have a higher level abstraction.
Also note the existence of FORTRAN language bindings, and the low level but persistent rumbles about C++ language bindings that have continued for the past 20 years or so.
Then note the number of languages where the doco tells you all about how "we wrap the C library", but what is actually there is in effect a language binding for POSIX for that language.
In terms of wrappers of POSIX, I'm definitely not saying that POSIX is dead and not used. It's great that it is a beginning point for porting to various platforms. What I was more pointing out is that as a developer, its your stdlib's portability that is key. For example, many languages use POSIX interfaces where they can, and then switch to platform specific interfaces where necessary, for performance or other reasons.
That's what I was going to say about UNIX, C, and POSIX. The UNIX and C languages' made tradeoffs specifically oriented around the weak hardware they had. We no longer need those. POSIX evolved from a bunch of competing vendors doing a mix of justifiable and weird stuff with a standard trying to abstract it all. It's inherently going to be worse than a clean design over the smaller number of modern UNIXen without all the politics that went into POSIX.
And then we have fact that many portable apps need to run on Windows, Mac, iPhone, etc POSIX ranged from useless to lacking platform-specific capabilities on these. So, a clean-slate design covering each of these platforms is the obvious solution. Fortunately, we have a bunch of them with varying styles and tradeoffs to choose from. :)
You're confusing popularity with being technically antiquated or obsolete.
If you're trying to write a portable app between all of those platforms, would you choose POSIX or would you instead pick a framework or language that has a better high level abstraction?
The thesis presented in the article is that POSIX is outdated. The popularity of a platform, particularly one which was forced upon the world through a de-facto monopoly, is entirely irrelevant in any argument on the technical merits of a technical standard, particularly when some of those platforms were designed with the express purpose of locking out virtually all standardized programming languages.
Technically speaking I think the standard is great, especially around having consistent definitions for thing like close(). But there are better options on all platforms to the select() and poll() interfaces.
In terms of my desires for open platforms that support open standards, I'm completely with you. If anything what we should be debating is how to update the POSIX standard such that it is worthwhile to target during development.
I don't see how it's helpful to ignore what is popular out in the world.
No, the point of POSIX is to specify a set of interfaces that software vendors should provide and target if their goal is to provide an UNIX variant or develop UNIX-compliant software.
The key issue is that the main requirement is willingness on behalf of the software vendors to comply with the standard, and offer/target the standardized interface.
If a software company is blatantly hostile regarding interoperability, and bases their business model on vendor lock-in strategies then it's quite obvious that those companies are particularly are not willing to play ball with others regarding interoperability.
Yet, somehow you're presenting those companies, and the consequences of the actions taken by companies that are openly hostile regarding interoperability, as some sort of proof that a specific interoperability target suffers from technical problems.
I'd say if you can run bash and gcc on Linux, OSX and BSD, it mostly succeeded.
User's probably won't notice, but they tend not to care about standards anyway.
You're confusing popularity with being outdated.
It's irrelevant if some vendors intentionallly go out of their way to avoid implementing any standard interface which, in the very least, can't be controlled by them.
And it's also absurd that you're quoting platforms which even restrict which programming language can be used to write applications purely due to their business strategies.
POSIX was supposed to be popular to the point of default way that apps were written. It failed. Whereas there's quite a bit of portable apps on (insert framework here). They succeeded with many running on non-UNIX OS's, too. You'll need those or similar things in your code instead of POSIX for truly, portable apps. That's his point.
If you read up on why companies such as Microsoft make it their point to break and avoid standards, you would understand why your comments regarding "popularity" are moot.
Business strategies don't make or break technical merits.
Microsoft's strategy was to create a permanent dependence on them for long-term profitability. They had too many tactics for that to cover here. Once open standards proliferated, they subverted them too with proprietary extensions under "Embrace, Extend, Extinguish" strategy. Companies buying into their solutions and using non-standard features paid the price.
Regardless, people still created apps that worked on such proprietary platforms and UNIXen via portability frameworks or libraries. Some products using them became quite popular (eg Apache, Netscape/Firefox) with some of the frameworks themselves becoming popular (eg Qt, GTK, wxWindows, Tk). POSIX's got into enough stuff that it was a default many picked up. Whereas, these others people went to willingly even when POSIX was available with even more portability. That's why they were objectively more successful on portability.
That point is moot, as those platforms were designed with the express purpose of being particularly hostile regarding interoperability while forcing other corporate products as alternatives, in a well-known lock-in strategy.
They were designed with the purpose of giving users a way to get things done that was profitable to the company. The first part along with lock-in techniques are why most of the desktop market is Windows with a large chunk of the server market. That means the kind of portability that matters to people wanting to maximize benefit to users or profits to the company better include a Windows version. There are portability alternatives to POSIX that allow that usually with other benefits on top of it. Far from moot...
No, they actually weren't.
The "users" don't have any say whether Windows complies with any standard, or even if Microsoft breaks all of them to try to force the world to submit to their "vendor lock-in" business strategy.
Trying to pass off the consequences of vendor lock-in policies as technical arguments is somewhere between absurd and disingenuous.
By what metric?
POSIX was never trying to be the universal interface for all kinds of applications, just some sane common interface for UNIX vendors, all of which do implement POSIX to a large extent these days.
You could perhaps make the argument that by the time POSIX saw enough adoption, software development evolved and that nowdays there are lots of interfaces that are not part of POSIX/not portable, but that's a slightly different issue.
Linux does it, it is me that doesn't understand, I see.
*BSD devs jump of joy having to port GNU/Linux applications that depend on D-BUS, systemd, or any other Linux specific APIs.
So POSIX was a long series of attempts to codify baseline Unix practice that arose because in the '87 time period there were tens of small, warring, incompatible Unix distributions ( cf. http://www.ugu.com/sui/ugu/show?ugu.flavors ), and systems software vendors could not write software that worked across all of these platforms properly.
The original post report is a cri de coeur call to action written in POSIX magazine attempting to oversell the story with a clickbait title in order to urge further POSIX work. This thread and others is loaded with people that have only read the title and have their own half-formed opinions.
There was never any idea that all apps everywhere would be written with POSIX. That's a ridiculous reading. I was there at the time. It was just an effort to get all of the Unix vendors on the same page so that the big commercial non-Unix OS vendor and their salespeople wouldn't eat the lunch of the high end workstation and server market.
And it succeeded phenomenally. With POSIX as a more-or-less stable substrate, it became possible to run, e.g., gnu's entire platform on almost anything that was POSIX-compliant or close; e.g. SunOS, Solaris, and eventually Linux, with minimal effort. It became possible for software like vi, emacs, mysql, sqlite to exist. Sysadmins could write shell scripts without anything like the kind of portability fear they had in the past.
Uncoincidentally, this continues today; all of the non-Windows software you're using is based on POSIX standards. Every phone is packed with POSIX-foundationed databases, daemons and sometimes even userlands. And of course there's Linux which, despite the continuing efforts of Red Hat, is about as POSIX as it comes.
Read the box at the bottom of page 9 of the original post. POSIX won, totally and unconditionally.
Precisely.
Additionallly, criticizing the technical merits of an international standard established in 1997 is a disingenuous ordeal. If anyone actually has any meaningful and tangible technical issue to point out regarding POSIX, they should simply put forth an implementation that actually addresses the issues and use it as a basis for a discussion to update the standard.
Standards are meant to be used as references, and therefore are meant to be updated when the need arises.
Far as early days, you might be able to help me out on a point I have few sources on. One history of Xenix posted here claimed its influence and design choices had strong influence on adoption of features that latter led to POSIX. Source was biased so hard to assess. What's your take on Xenix's influence on adoption of standard or POSIX-like API's?
That said, POSIX was never designed for general portability. Just as a seal of unifying approval and sanity for existing practice; far more descriptive than prescriptive.
1 "usage is driven by high-level frameworks, which impacts POSIX’s portability goals.", that's the libraries/frameworks fault, plus have they heard of GCC? It works on many OS's.
2 "extension APIs, namely ioctl, dominate modern POSIX usage patterns as OS developers increasingly use them to build support for abstractions missing from the POSIX standard" Perhaps they have missed the point, that without the POSIX standard these extensions would not exist in the first place. They might just complain about procedural coding practices being outdated when comparing to OOP coding practices, even though OOP brings slow bloat to the table as well. Perhaps a better question to ask is, why has a framework/library got some functionality which could have used some of the POSIX api's but didn't? Was it expedient to keep it within the framework/library and bypass the Posix API's altogether?
3."new abstractions are arising, driven by the same POSIX limitations across the three OSes, but the new abstractions are not converging." So abstractions are a new name for libraries and framework and not the concept like OOP coding is an abstraction. Perhaps complaining to the authors of libraries and frameworks that don't port would be better? POSIX is like a foundation that allows creativity to take place. This is why we see the differences in libs/frameworks, different OS's have different parameters, different design ethos or objectives. Even in Windows, some API's are used more frequently than others, for example I cant remember when I last used the API's involved in getting a dial up modem to work, or using API's to print direct to a dot matrix printer. This paper just seems to be an analysis of API's not used as frequently as they used to be, sometimes because libraries/frameworks have rightly or wrongly replaced them for a myriad of reasons.
Qudos to them for downloading so many apps and analysing them though. Just what you need if you are looking for backdoors en-masse or exploitable bugs.