Transcending Posix: The End of an Era?
usenix.org
usenix.org
I'm seeing an increasing trend of pushback against this norm. An early example was David Crawshaw's one-process programming notes [1]. Running the database in the same process as the application server, using SQLite, is getting more popular with the rise of Litestream [2]. Earlier this year, I found the post "One machine can go pretty far if you build things properly" [3] quite refreshing.
Most of us can ignore FAANG-scale problems and keep right on using POSIX on a handful of machines.
But his architecture does seem to be consistent with a "minutes of downtime" model. He's using AWS, and has his database on a separate EBS volume with a sane backup strategy. So he's not manually fixing servers, and has reasonable migration routes for most disaster scenarios.
Except for PBKAC, which is what really kills most servers. And HA servers are more vulnerable to that, since they're more complicated.
I posit that this kind of "deal killer" is most often a wish list item and not a true need. I think most teams without a working product think these kinds of theoretical reliability issues are "deal killers" as a form of premature optimization.
I worked at a FANG doing a product where we thought availability issues caused by sessions being "owned" by a single server design was a deal killer. I.e. that one machine could crash at any time and people would notice, we thought. We spent a lot of time designing a fancy fully distributed system where sessions could migrate seamlessly, etc. Spent the good part of a year designing and implementing it.
Then, before we finished, a PM orchestrated purchase of a startup that had a launched product with similar functionality. Its design held per-user session state on a single server and was thus much simpler. It was almost laughably simple compared to what we were attempting. The kind of design you'd write on a napkin over a burrito lunch as minimally viable, and quickly code up -- just what you'd do in a startup.
After the acquisition we had big arguments between our team and those at the startup about which core technology the FANG should go forward with. We'd point at math and theory about availability and failure rates. They'd point at happy users and a working product. It ended with a VP pointing at the startup's launched product saying "we're going with what is working now." Within months the product was working within the FANG's production infrastructure, and it has run almost unchanged architecturally for over a decade. Is the system theoretically less reliable than our fancier would-be system? Yes. Does anybody actually notice or care? No.
That is not at all to say that it is a deal breaker for everyone, but it certainly will be for some companies.
One trend in OS research (what little exists) is the idea of the database OS. Taking that as an inspiration I think there's a better way to structure things to get that same simplicity and in fact even more, but without many of the downsides. I'm planning to write about it more at some point on my company blog (https://hydraulic.software/blog.html) but here's a quick summary. See what you think.
---
In a traditional 3-tier CRUD web app you have the RDBMS, then stateless web servers, then JavaScript and HTML in the browser running a pseudo-stateless app. Because browsers don't understand load balancing you probably also have an LB in there so you can scale and upgrade the web server layer without user-visible downtime. The JS/HTML speaks an app specific ad-hoc RPC protocol that represents RPCs as document fetches, and your web server (mostly) translates back and forth between this protocol and whatever protocol your RDBMS speaks layering access control on top (because the RDBMS doesn't know who is logged in).
This approach is standard and lets people use web browsers which have some advantages, but creates numerous problems. It's complex, expensive, limiting for the end user, every app requires large amounts of boilerplate glue code, and it's extremely error prone. XSS, XSRF and SQL injection are all bugs that are created by this choice of architecture.
These problems can be fixed by using "two tier architecture". In two tier architecture you have your RDBMS cluster directly exposed to end users, and users log in directly to their RDBMS account using an app. The app ships the full database driver and uses it to obtain RPC services. Ordinary CRUD/ACL logic can be done with common SQL features like views, stored procedures and row level security [1][2][3]. Any server-side code that isn't neatly expressible with SQL is implemented as RDBMS server plugins.
At a stroke this architecture solves the following problems:
1. SQL injection bugs disappear by design because the RDBMS enforces security, not a highly privileged web app. By implication you can happily give power users like business analysts direct SQL query access to do obscure/one-off things that might otherwise turn into abandoned backlog items.
2. XSS, XSRF and all the other escaping bugs go away, because you're not writing a web app anymore - data is pulled straight from the database's binary protocol into your UI toolkit's data structures. Buffer lengths are signalled OOB across the entire stack.
3. You don't need a hardware/DNS load balancer anymore because good DB drivers can do client-side load balancing.
4. You don't need to design ad-hoc JSON/REST protocols that e.g. frequently suck at pagination, because you can just invoke server-side procedures directly. The DB takes care of serialization, result streaming, type safety, access control, error reporting and more.
5. The protocol gives you batching for free, so if you have some server logic written in e.g. JavaScript, Python, Kotlin, Java etc then it can easily use query results as input or output and you can control latency costs. With some databases like PostgreSQL you get server push/notifications.
6. You can use whatever libraries and programming languages you want.
This architecture lacks popularity today because to make it viable you need a few things that weren't available until very recently (and a few useful things still aren't yet). At minimum:
1. You need a way to distribute and update GUI desktop apps that isn't incredibly painful, ideally one that works well with JVM apps because JDBC drivers tend to have lots of features. Enter my new company, stage left (yes! that's right! this whole comment is a giant ad for our product). Hydraulic Conveyor was launched in July and makes distributing and updating desktop apps as easy as with a web app [4].
2. You're more dependent on having a good RDBMS. PostgreSQL only got RLS recently and needs extra software to scale client connections well. MS SQL Server is better but some devs would feel "weird" buying a database (it's not that expensive though). Hosted DBs usually don't let you install arbitrary extensions.
3. You need solid UI toolkits with modern themes. JetBrains has ported the new Android UI toolkit to the desktop [5] allowing lots of code sharing. It's reactive and thus has a Kotlin language dependency. JavaFX is a more traditional OOP toolkit with CSS support, good business widgets and is accessible from more languages for those who prefer that; it also now has a modern GitHub-inspired SASS based style pack that looks great [6] (grab the sampler app here [7]). For Lispers there's a reactive layer over the top [8].
4. There's some smaller tools that would be useful e.g. for letting you log into your DB with OAuth, for ensuring DB traffic can get through proxies.
Downsides?
1. Migrating between DB vendors is maybe harder. Though, the moment you have >1 web server you have the problem of doing a 'live' migration anyway, so the issues aren't fundamentally different, it'd just take longer.
2. Users have install your app. That's not hard and in a managed IT environment the apps can be pushed out centrally. Developers often get hung up on this point but the success of the installed app model on mobile, popularity of Electron and the whole video game industry shows users don't actually care much, as long as they plan to use the app regularly.
3. To do mobile/tablet you'd want to ship the DB driver as part of your app. There might be oddities involved, though in theory JDBC drivers could run on Android and be compiled to native for iOS using GraalVM.
4. Skills, hiring, etc. You'd want more senior devs to trailblaze this first before asking juniors to learn it.
[1] https://www.postgresql.org/docs/current/ddl-rowsecurity.html
[2] https://docs.microsoft.com/en-us/sql/relational-databases/se...
[3] https://docs.oracle.com/database/121/TDPSG/GUID-72D524FF-5A8...
[4] https://hydraulic.software/
[5] https://www.jetbrains.com/lp/compose-mpp/
[6] https://github.com/mkpaz/atlantafx
[7] https://downloads.hydraulic.dev/atlantafx/sampler/download.h...
BTW, Conveyor is free for open source projects, and currently free for commercial too. The current versions focus on solving the basics really well. Deployment/update is like for a website statically generated from Markdown. You can build fully signed-or-self-signed and self-updating packages from cross-platform artifacts on whatever OS you happen to use with one command. So, you can package up Electron and JVM apps given just inputs like JS or JAR files, from your dev laptop or Linux CI box, and you get packages for every OS. You also get a simple download HTML page that detects the user's OS and CPU.
To do a new release you just re-build and re-upload the site. Clients will start updating immediately. On macOS it uses Sparkle, on Windows the OS will do the updates in the background and Linux users get packages.
It does native apps too but then of course you need to compile the binaries for each OS yourself.
One possibility we're researching is to page code in from the database on demand. That way you only have to push updates to the client occasionally, like when refreshing the core runtimes. For changing business logic the client would use SQL queries to speculatively load code based on what other clients have been requesting. If it works it means you can get rid of minification, bundling, all the other hacks web devs do to reduce requests and round-tripping, whilst keeping the "instant" deployment browsers give you.
Careful schema design to support synchronization without collisions is the only real difference between this kind of app and CRUD apps that expect to always be able to reach the Internet.
I’m trying to understand more real world examples of syncing offline with online.
Thanks!
Basically, my schema design prohibits use of UPDATE and requires that every row have a timestamp. The clients maintain a synchronization log to ensure they have fetched every available row. The keep track of which rows have not yet been sent up to the server.
This means that finding the current state of things that can change means doing a "SELECT column ORDER by timestamp DESC LIMIT 1" in order to see the latest state and always doing INSERT instead of UPDATE to update state.
In some cases, I am storing a delta in a row instead of a complete state representation. This means that some views have to replay the changes to show the current state. I cache the result of these.
I do some general high level caching on the client side to make all of this as fast as possible. I have watchdogs set on the local GUI to warn me when latency of the GUI event loop is over 200 milliseconds. I use these warnings to focus effort on caching and other optimizations.
There were other issues of course, it wasn't just about distribution. Too bad so many downsides came along with the upsides. The real goal for OS research should be IMHO to find ways to combine what people like about web dev with what people like about desktop/mobile dev. All the action is above the POSIX layer.
How do you suggest achieving this in desktop apps? Some kind of log streaming?
What do you think drove this? Presumably plenty of people in the dark mass of 9-to-5 devs were happy with VB/Delphi. Jonathan Edwards has written [1] that VB came from "a more civilized age. Before the dark times… before the web." Did nerdy devs like me, with our adolescent anti-Microsoft attitude (speaking for myself anyway; I was born in 1980), ruin everything?
Delphi was good but a compiled language with manual memory management. It was very easy to write code that crashed, which would nuke the user's state leaving little evidence of what happened. It also had a lot of legacy quirks like not allowing circular dependencies between compilation units, networking support was poor (iirc TCP, HTTP and other classes required you to buy a third party library). The VCL was a wrapper around Win32, which had some great strengths but also really frustrating weaknesses e.g. extremely poor/non-existent layout management, poor support for typography and no support for styling or branding. There were many positive aspects of course.
Microsoft gave you VB or C++, both with Win32 again. The C++ developer experience was horrid. VB was at least a scripting language with garbage collection, but, it was also constrained by the fact that B stood for "Beginners" so Microsoft were very reluctant to fix any of its legacy or add more powerful features.
Compared to that situation, scripting languages and especially Perl had some massive advantages:
1. Ran on UNIX/Big Iron which is where all the best hardware and databases could be found. Lots of devs liked UNIX because it was CLI and developer oriented.
2. Unashamedly designed for experts with tons of powerful features, quality of life stuff like integrated regex, garbage collection, proper stack traces, error logs you could view via telnet within seconds etc.
2. CPAN provided an ever growing repository of open source libraries, instantly accessible, for free! On Windows there were very few libraries, they were mostly quite expensive and closed source, no easy way to discover them (pre-Google) and only C++/Delphi devs could write them. VB was sort of a consumer-only language. Open source culture started with RMS at the MIT AI lab and so was very UNIX centric for a long time. Arguably it still is.
Really, it's hard to overstate how revolutionary proper garbage collection + CPAN was. GC is a force multiplier and CPAN is the granddaddy of all the open source library repositories we take for granted today. Imagine how unproductive you'd be without them.
The big downside was that Perl had no UI libraries and didn't really run on Windows. So how do you use it to write apps for normal people? Then Netscape started adding interactivity features to the web and it was all totally text based. Text was Perl's forte! Add the <form> tag, CGI, HTTP and now you're cooking with gas. Bye bye hateful 0xC00005 Access Violation errors and useless bug reports like "I clicked a button and the app disappeared".
The web was a huge step back for users, who went from having pretty sophisticated GUIs with fast table views, menus, shortcut keys, context menus, Office integration, working copy/paste, instant UI response etc to ... well, the web. But often users will suffer through that if it makes their devs more productive because all they really care about are features and competitive advantage. The web was crude but it let people escape the Windows ecosystem to one with open source + GC + pro languages + big iron UNIX.
There is revisionist history, and then there is that statement. That statement ... is almost offensive.
... Why did the "golden age" end? Because Microsoft sucked in so so so so so so so many ways. That statement is a bald faced lie, a whitewashing attempt to forgive Microsoft from inflicting millions of man years in damage to the IT industry over three decades of anticompetitive practices to keep their substandard software in a position of market dominance.
But for anyone beyond the MCSE factory programmer (aka NOT the thought leaders of the industry) aside from those profiting handsomely from the Microsoft evil empire, did not like Microsoft.
In the days of DOS, you had DOS or UNIX. Which was better?
In the days of Windows in the 3.1 and even pretty much in the windows 95, you didn't have preemptive multitasking (something that non-windows had for 20 years at that point). It crashed CONSTANTLY, had no security, required restarts for practically anything that was installed.
Meanwhile the UNIX people would brag about machines not being rebooted for years. This was before the constant patch cycle was a thing.
Microsoft's apis were slapdash and disorganized, and frequently went out of favor.
During this time Microsoft would constantly crush and screw over competitors and engage in constant anticompetitive behavior. If you didn't suck on the teat of Microsoft, your platform was under unrelenting assault, not by pure technical achievement, but by the full smorgasborg of corporate dirty tricks, FUD, bribery/kickbacks, lobbying, lies, secret apis, golf schmoozing with nontechnical CIOs to close deals, no or undermined standards, etc.
The graveyard is long: Sybase, Lotus 123, Netscape, Novell.
The Microsoft times were a time of being stuck with one option: and OS that crashed constantly or is utterly porous to attackers. A browser that has incompatible web apis and a disbanded developer team (IE 6) that gets corporate mandated and is a thorn in your side in the entire IT stack for two decades. Databases stolen from companies (Sybase signed SUCH a bad deal, it astonishes me to this day) running on platforms that can't stay up. Office software with inaccessible file formats and byzantine and closed programmatic apis for accessing it. A substandard desktop UI.
If you used Microsoft software with an ounce of historical knowledge or awareness, you could see the prison. You had no practical choices. All the executives in your company were bought and paid for. Microsoft had sales forces that tracked non-Microsoft systems and targeted them within companies by any means necessary. Every new piece of software written in MIcrosoft had to pay the "sociopath management tax" and go through extensive reviews on how it could be used to further or maintain Microsoft's empire and control.
Their software was deliberately dumped in beta form onto the market to crowd out the competitors.
None of this is an "adolescent" attitude. I'll repeat myself: MILLIONS OF MAN HOURS OF DAMAGE. You know, probably billions. Decades x tens of millions of workers.
This isn't just IT programmer frustration. This is bad applications forced on non-programmer users. This is better companies, better software, better IT industry denied proper funding and profits. Instead, Microsoft took trillions of dollars in revenues from them. This is undermining a free market, free ideas, and freedom for Microsoft's profit.
• Until Linux, UNIX didn't even run on PCs at all, only expensive proprietary hardware. Windows ran on cheap machines made by a competitive market. Businesses wanted PCs for lots of reasons (e.g. Office).
• UNIX wasn't a single OS, it was many forks that were not compatible. You couldn't just write an app for UNIX, put it on CDs and sell it.
• Anything GUI related was terrible compared to Win32. Motif was the standard but cave man tech compared to what you could do on the client with Windows.
• UNIX was wildly more expensive.
• PCs had a thriving gaming culture. It meant people used the same tech at home as at work, creating a self-training workforce. UNIX vendors didn't care about games, they were too 'serious' for that. Their loss.
• Windows 95 did do pre-emptive multi-tasking, by the way. Only Win16 apps that hadn't been ported were cooperatively multi-tasked. Those apps didn't stick around very long because porting to Win32 was easy and Windows 95 was an absolute monster success. People queued up at midnight around the blocks of PC stores to buy it, it was so popular.
• Windows apps crashed a lot compared to UNIX apps because the average Windows machine ran way more apps than the average UNIX machine, ran on way lower quality hardware that was often flaky, had way more diversity of peripherals and configurations, and apps were deployed in places where it was hard to get crash logs (no networks).
• Windows machines didn't have uptimes of years because nobody cared. They were workstations. You turned them off at the end of the day when you went home because otherwise they'd burn their CRTs in and their lifetime was reduced. The culture of leaving non-server machines on all the time and never rebooting them only began once ACPI and other power management tech started to become common (something non-Apple UNIX struggles with to this day). And once it was useful to do so, Microsoft had a version of Windows that could have uptimes of months or years, no big deal.
Not true. There was Xenix, SCO, and Coherent as 3 examples off the top of my head.
Here it is: https://archive.org/details/CoherentMan
And all things being equal you could still get OS/2 as low as $49,
> The suggested introductory price of OS/2 2.0 is $139. However, the cost falls to $99 for users upgrading from DOS, which includes just about anyone, and to $49 for users who are willing to turn in their Microsoft version of Windows.
https://www.nytimes.com/1992/04/21/science/personal-computer...
Also, VMS and other OSes were the ones that ran for years without rebooting. Unix at the time was not so stable. Before Tcl, John Ousterhout wrote a log-structured file system for Unix because their Unix systems crashed often enough and took so long to boot that a possible loss of data but fast boot-up was deemed better than the lengthier downtimes with the existing file system.
So the PC market went with Microsoft and its encompassing environment, much to everyone's detriment. Fortunately, we've all moved on to the web and JavaScript and everything is now sunshine and roses. :-)
Bad management, bad products, not willing to reduce prices.
You see it nowadays on the Linux Desktop, instead of uniting, everyone goes do their own little thing.
No wonder it doesn't work out.
Linux won on the server room, thanks to being a cheap UNIX clone, and now with cloud computing and managed language runtimes, it hardly matters if it is there, or they are running on top of a type-1 hypervisor.
Web applications can come close to directly accessing the database by using GraphQL with something like Hasura or PostGraphile. PostGraphile even uses Postgres's row-level security. A colleague and I once did a project using Hasura with a SPA-style JavaScript front-end and a separate back-end service driven by Hasura webhooks for doing the actual computation, and we ended up being unhappy with that approach. Some of our problems were related to the SPA architecture, but some were related to our use of GraphQL and Hasura.
We ended up starting over with a typical server-rendered web application, where the server itself accesses the database and communicates with the computation-heavy back-end service over gRPC, using a minimum of client-side JavaScript. I remain happy with that architecture, though I continue to explore different ways of integrating modest amounts of client-side JavaScript for interactivity and real-time updates. And to bring it back to the topic of my previous comment, if we assume that there has to be a server rendering HTML anyway, then I think it often makes sense to reduce complexity by bringing the database into that server process. I haven't yet tried that in production, though.
I think my preference is to use HTTP primarily as it was originally intended, for fetching HTML, as well as for form submissions. For finer-grained interactivity, I think it's better to use WebSockets as opposed to REST-ish requests and responses. I'm not dogmatic on that, though.
On web apps versus packaged desktop apps, I'm still inclined to go with web whenever feasible, and only develop a packaged desktop app if a web app just won't work. Being able to use an application without an installation step is really powerful for pre-sales demos or trials, for onboarding new users, and for applications that may only be used occasionally by any given user. Even for an application that you use all the time, a web app can be fine, as the popularity of Google Docs demonstrates. For example, if you just want to get the browser chrome out of the way, desktop browsers support "installing" a web app as an OS-level app with no browser chrome. IMO, Hydraulic's Eton demo app could just as well be a web app.
I look forward to your blog post, though even your preliminary HN comment offers a lot to think about.
Yes, for transient apps that are only used occasionally or where the user isn't committed e.g. social networks, the combination of sandboxing + no integration/unintegration steps, automatic deletion from the cache etc is really useful. Of course there's no rule that says only web browsers can supply these features. Other kinds of browser-like thing can do so too.
It's also worth thinking a bit outside the box. Although we claim web apps don't have installation steps, that's not really true most of the time. The lack of any explicit integration step ("install") means the browser doesn't know if the user values anything they do on the site. So you have to provide data persistence, and that in turn means you need a signup / account creation flow. It also means you're on the hook to store, replicate and back up any data any user ever creates, even if they only used it once and then never return.
Well, a lot of users really hate creating yet another account, especially if they aren't committed yet. It's tedious, they think you'll spam them with marketing emails and they're probably right, plus they don't want to make another password. Or they make one, abandon for a year, then can't remember how to log in.
You might think that's so fundamental it can't be any other way, but it's really just a side effect of how browsers have evolved. Think about how you might do demos outside the browser. You could just have a trial mode where the app spins up a local in-process RDBMS like H2 that writes the data into the app's private data area (on Windows) or home directory on macOS/Linux. No accounts necessary - just one or two clicks to download+trigger app install, and you're done. If/when the user decides to graduate, then they create an account and the app uploads the data from the local DB to the real remote DB. If they abandon it and don't care, it's not your problem, it costs you nothing. If they run low on disk space the OS will suggest they delete old unused apps at that point and you'll get tossed along with the rest.
Mostly though, this is about making developers more productive. If the primary determinant of your success is feature throughput and not shaving a few seconds off your onboarding, e.g. you're making specialised apps, internal apps, enterprise software, then optimizing for better dev environments can make sense. Installation just isn't that bad.
First, simply, I don't know anyone that puts their DB connections "on the internet". That live, raw, database "SQL" socket to be poked, prodded, hammered, and cajoled by complete strangers (or their automatronic minions).
Second, is DB latency. Sending "coarse" grained service requests across the planet is one thing compared to the potential volume of random SQL commands that folks do.
Mind, much of that can be mitigated if you build a stored procedure service layer. But classic "2 tier" "client/server" work didn't do that exclusively, just just threw out SQL willy nilly as the need dictated.
As old school as I am, even I tend to shy away from the "monolithic DB". You think your app was a monolith before, wait until it's all baked into Oracle. I've always found the DB to be a "bad citizen" when it comes to things like versioning, source code control, etc.
Even if I were doing a desktop app, I still think I would prefer a middle tier ("application server") managing service endpoints than cram it all into the DB, especially today.
Latency is easy to screw up whether you do web apps or direct SQL connections. You have to be conscious of what a request costs, and you can easily batch SQL queries. Yes, you have to watch out for frameworks that spam the DB but those are bad news anyway, and of course there are lots of web frameworks that generate inefficient code. Not sure it's so different.
Your app will have to deal with DB versioning whether it's a web app or not. Tools like Flyway help a lot with linking your DB to version control and CI.
Nonetheless, I totally understand where you're coming from. Thanks for the thoughts.
A few? I'd say most are accidental and the rest are just bad ideas...
>But people do it and the sky doesn't fall.
Well, the same is true for playing Russian roulette too. Most of the times you're winning!
One concern I can think of is NGINX might have better DoS protection. What else do you have in mind?
I mean smooth async, fuzzy matching and reactive sub trees. I'm not a modern fanatic, I actually enjoy the good old as400 or win311 model a lot, and use old emacs daily. But there was a big chunk of the 90s where GUIs were really really subpar.
Generally if you can do it on mobile you can do it elsewhere, right? If you want something like ReactJS and coroutines/async/await, look at Jetpack Compose. It's inspired by ReactJS but for Android/Desktop: https://developer.android.com/jetpack/compose
You don't need any particular UI toolkit though. Many years ago I did a tutorial on "functional programming in Kotlin":
https://www.youtube.com/watch?v=AhA-Q7MOre0
It uses JavaFX with a library called ReactFX that adds functional utilities on top of the UI framework. It shows how to do async fuzzy matching of user input against a large set of ngrams. I guess that's in the region of what you mean too.
nothing I said was web only, but web has focused on it and made it basic lingo; I admit not being aware of the javafx world evolution when I used it it was crude (circa 2010).
reactive subtrees ~= two way data binding over dom like trees, i don't recall running into such ui rendering models before
https://openjfx.io/javadoc/18/javafx.base/javafx/beans/bindi...
If a two-tier app sends out emails, PLSQL/dbplugin does it. Now every ops task for sending emails involves the DB and by extension, your data is at stake. To launch a new parallel process, or to roll a new version, or to spread to a different location, or to kill a frozen process, or to measure much RAM a new feature has eaten, these are all DB tasks despite the fact that the task was just for a send-email feature.
Anything happening server-side (i.e. not on a user's device) needs to pass DBA middlepersons.
To put it back on feet, the architecture might be: the DB is one of the services. A frontend can talk to a database, and the two can work out the protocol, the authn/authz, the load balancing. They don't need any CRUD "backend" that is not really a "back" "end" but just a glorified boilerplate SQL-to-JSON converter.
The tradeoff is that you lose a lot of implicit trust. An email service cannot trust the frontend with the business rules. If user is allowed to only send to a set of recipients - it's an email service that needs to query that set from the DB.
If you wanted isolation of an email service in this design, you'd use the DB as a task queue. The app triggers a procedure (written in SQL, Python, Java or whatever) which verifies the business logic and then does an insert to a tasks table. The email microservice wakes up and processes the queued tasks. That's a pretty common design already.
I thought about the whole idea over the weekend a bit and I'd say it is worth a try.
If you say that you have the distribution problem figured out, that makes it viable, it was the biggest obstacle in the 90s. What I'd expect it to mean is that to roll out a significant DB change, the frontend can self-update and not lose a hour-worth of users' unsaved work.
Also I think when selling this, you don't need to avoid the Delphi nostalgia that much. Everyone old who sees "remove the middle tier" will instantly go into mental mode of "uh-oh, those who do not learn from history are bound to repeat it". You are seeing a lot of it around this subthread - if you acknowledge upfront that you know you build on that past exp, it adds credibility.
People always have different thresholds for what "solved" means. Today Conveyor gives you a Chrome-style update experience on Windows where the app updates in the background even if it's being used. The user isn't disrupted. Ditto on macOS if the user states they want silent background updates at first update (we'll probably tweak this so the user isn't asked and must opt-out explicitly). The user won't lose any unsaved work or be surprised by sudden changes.
So to make a DB change, you need to do it in a backwards compatible way until the clients have had a chance to fully update. Probably that means being compatible for a few days, if you have users who aren't always online. This is probably solved enough for the general case.
The developer experience in that mode is exactly the same as compiling Markdown to HTML using Jekyll or Hugo.
The next step is to go further, so code changes can take effect much faster than a background update cycle. It requires the client to have some notion of a page, screen, activity etc - some point between interactions where you can make changes without disrupting the user. And it requires the client to be informed by the server if code has changed, even whilst the user is running the app. This takes more work, but it's on the roadmap. Mostly we think the async model is OK. You have to change your schemas in backwards compatible ways even in the web case to avoid site outages or different web servers getting confused during a rolling upgrade.
As someone who uses SQLite a lot, I'm suspicious of this claim. Litestream is strictly a backup tool, or, as its author puts it, disaster recovery tool. It gives you a bit more peace of mind than good old periodic snapshots, but it does not give you actual usable replication,* so I doubt it meaningfully increased SQLite adoption in the RDBMS space (compared to the application data format space where it has always done well).
* There was a live read replica beta which has since been dropped. Author did mention a separate tool they're working on which will include live replication. https://github.com/benbjohnson/litestream/issues/8#issuecomm...
[0]: https://github.com/superfly/litefs
But what about global customers? Most of the planet just eats the latency? What about single node failure? You usually need to scale past n=1 for a public facing service. It's not just about Google scale.
If your ui maintains state via some kind of async layer then the latency might not be observable at all.
For a lot of "embedded" use cases like robotics, where you are trying to squeeze out maximum performance from a single machine (the hardware you have on the robot), POSIX is and will remain a hindrance.
This is of course far from true; CTSS and especially ITS (which was so programmer friendly that rather than what is called today a “shell”, it’s user interface was a debugger) long predated it and both Thompson and Ritchie were exposed to them while at MIT to work on Multics. The supreme programming environments for programmers, the MIT Lispms and the D-machines at PARC are the ultimate in programmer-friendly OSes that are, for various important reasons, unmatched today.
Also, in their early papers about Unix they talked about how they didn’t need all that fancy stuff that mainframe systems like Multics needed, like VM and IO modules (Unix just does everything through the kernel in a simple fashion). Well, it was for a small computer system. There were good reasons that mainframes were structured the way they were and by the mid 80s that model started migrating to workstations (e.g. Britton-Lee accelerator and graphics subsystems) and by the 90s that model was already quite common in microcomputers too (for example with mpus in disk controllers and network interfaces).
But the Unix cultists ignored that revolution. They came late to the party with regards to networking; even when developing the sockets interface they relied heavily on the PDP-10 approach, but seemingly didn’t learn anything else from that.
Glad they are finally noticing that the world has changed.
Had it been a commercial product from the get go, and history would have gone quite differently.
Now the posix APIs feels like the worst of both worlds.
Most applications shouldn't really store data on the filesystem, and probably not even configuration. Most applications use a database for a zillion reasons (e.g. better backup, restore, consistency, replication, transactions, avoiding random writes, ...). Yet the filesystem is also not a great interface if you're building a database: you want more control over flushing and buffering and atomic file operations. The filesystem is mainly good at storing compiled code.
The "everything is a text file" interface is not great anymore, either. Simple manipulation of the output of basic commands (like text 'ps') requires error-prone parsing and quite some skill with things like grep, sed, awk, sort, etc. That's OK for experienced sysadmins working on individual machines, but doesn't integrate very well in larger systems where you have lots of automated access to that information. I have some experience doing sysadmin-like work, but I still run into problems with simple things like spaces in filenames and getting the quoting and escaping right in a regex. One option is for it to be more database-like, but that's not the only option. Erlang provides nice programmatic access to processes and other system information while still feeling very free and ad-hoc.
Of course, Linux is still great in many ways. But it feels like some of the core posix APIs and interfaces just aren't a great fit, and a lot of the interesting stuff is happening in non-posix APIs. And applications/services are being built on higher-level abstractions that aren't tied to single machines.
I think, like anything, it depends on what you're doing, and how you're doing it.
> The "everything is a text file" interface is not great anymore, either.
Text doesn't seem markedly different from a JSON API? Perhaps "worse is better" more often when you are composing several programs/apps to make one system? Even complex distributed systems.
> But it feels like some of the core posix APIs and interfaces just aren't a great fit
And I think these APIs can thrive on top of (the good parts of) POSIX. I'm not so certain we need to fundamentally rethink this layer, because this layer seems to work pretty well at what it does well. It should be pruned occasionally, but I'm not sure we need a whole new abstraction.
FWIW, I think this what the article is saying. Let's create some new interfaces where the old interfaces don't model the hardware well[0].
I thought it was an interesting idea until I actually got around to experimenting with Powershell recently. The ergonomics are more akin to Python than any of the GNU tools. I can’t help but wonder why not just use a language like Python rather than a heavyweight scripting language if it’s a similar LOE (speaking in general, I’m aware of CLR)? The benefit of the text based interface to me is the expediency of performing one-off CLI actions. I inclined think an object interface actually impedes this goal.
Json would be terrible as a primitive.
I was imagining something more like erlang, which is very OS-like. The types are very permissive/fleixble, and there's lots of ways to transform and manipulate results or feed it into something else without feeling code-heavy. It still feels comfortable to do ad-hoc stuff. The advantage is that in erlang you usually don't worry about basic stuff like record and field separators.
"Perhaps "worse is better" more often when you are composing several programs/apps to make one system?"
That's probably true as you zoom out. But even basic utilities in posix within the same system require a bunch of parsing. Imagine if a function call in a programming language required an intermediate text format. That's how unix feels sometimes.
Is it time to look at database-first operating systems again? There have been a few. Tandem's Guardian was very well regarded in its day. Unfortunately, Tandem was acquired by DEC/Compaq, which tried to move it to the Alpha around 1997. Then, after HP acquired what was left of Compaq, HP tried to move it to Itanium. (There was a MIPS version in there somewhere.) After all those bad decisions, in 2014 it was finally ported to x86, by which time few cared. There's still an HP offering.[1]
In Guardian, the database system owned the raw disks. There were no "files". If you wanted something that looked like a file, it was just a blob in a database entry. Everything was transaction-oriented, like SQL. That was the whole point. Everything was a redundant, recoverable transaction.
The OS was cluster oriented. Adding or replacing servers was normally done without shutdown. Databases were replicated, and you could add new disks, wait for them to come into sync, and remove the old ones.
All this in the 1970s. They were way ahead of their time.
- Direct disk access bypassing kernel fs, check.
- POSIX-like API standard, check. (ANSI SQL)
- Structured data storage with ACLs, check.
- Volume management, clustering, replication, RAID. Check.
- "Everything is a $FOO" philosophy, check. Where POSIX has files, RDBMS has tables.
- CLI/shell, check.
- Installable apps, check. (you can create stored procs by INSERTing java/javascript/etc into system tables).
Probably more parallels that don't occur right now. Lots of companies use the DB as their primary OS with the underlying kernel relegated to starting up the DB and configuring the hardware.
> Nevertheless, the cloud and serverless platforms are now facing a problem that operating systems had before POSIX: their APIs are fragmented and platform-specific, making it hard to write portable applications. Furthermore, these APIs are still largely CPU-centric, which makes it hard to efficiently utilize special-purpose accelerators and disaggregated hardware without resorting to custom solutions. For example, JavaScript is arguably in a similar position today as POSIX was in the past: it decouples the application logic from the underlying operating system and machine architecture. However, the JavaScript runtime is still CPU-centric, which makes it hard to offload parts of a JavaScript application to run on accelerators on the NIC or storage devices. Specifically, we need a language to express application logic that enables compilers and language run times to efficiently exploit the capabilities of the plethora of hardware resources emerging across different parts of the hardware stack. At the same time, it would be an an interesting thought experiment to ponder how different would the hardware design of these devices be without the CPU-centrism in POSIX.
It sounds like the author discovered the motivation behind the last time the POSIX standard was obsoleted, and some of the reasons why Plan9 was developed.
I've often lamented that I was born too late to watch Unix/POSIX be born and grow up. That's always felt like a golden time in computing. I also wasn't in the industry yet for the early 2000s and the rise of web-scale, though I was at least aware. It's exciting to think about what systems will do over the next twenty years. I'm looking forward to it.
TFA says "Therefore, we argue that the POSIX era is over, and future designs need to transcend POSIX." So they're actually not answering their own question, they're saying "the answer should be x" which is subtly different. There are a lot of should-be's in computing.
Ironically, while I am an heavy email user, at work it is mostly used for internal newsletters, all the communication happens over Slack.
slack replaced your internal, HR monitored, internal IM. Not email. Just so happens that lots of places were using email as their IM solution.
In case you missed part of my comment I mention the small detail of being an avid email user, but of course you needed to do some tiktok marketing.
For a lot of these services email is optional, and even with an email they won't let you sign up without confirming a mobile phone number.
In a modern smartphone, the OS directly controls just one of a dozen chips. The rest are running their own OSes which do their own thing independently of the main OS.
https://www.youtube.com/watch?v=36myc8wQhLo
Roscoe's talk is fairly long (the video is > 1h), but the basic thesis was taken up in a segment of Bryan Cantrill's 20 min OSFF talk:
https://www.youtube.com/watch?v=XbBzSSvT_P0
Both talks are very good; I recommend watching both, in either order.
For instance event based programming, "epoll" should replace "select". Synchronous programming, signalfd, timerfd, etc... but then "signals" should be more accurately classified, for instance in monothreaded applications segfault won't be delivered from a "signalfd" in a "epoll" syscall... and why not keep only the "realtime signal" behavior?
If POSIX only "add" new "options", in the end, you'll get gigantic spec just being the mirror image of the very few implementations since its size will make it unreasonable for a new implementation from scratch.
The "leaner POSIX" route seems the right way here, but it is really easier to say than to do.
fork() should be ditched.
Why? In favor of what?
1. Call a function that creates an empty child process in a suspended state.
2. Call functions that: map/write memory into the child process’s address space; add file descriptors to the child process; set the child process’s initial register values; and so on.
3. Call a function to unsuspend and run the child process.
My point was that embryonic processes aren't really the right solution since they require exposing a whole bunch to powerful primitives like read/write of remote address spaces in order to spawn processes. That ends up being slow (because each one of those calls is necessarily a syscall and requires manipulating page tables to mess with the other process). It also means to prevent abuse you need to be very carefully control when to revoke those privileges.
The obvious solution is to take all the operations you would have done to the remote process, encode them in some form, and pass them off to a secure agent (in this case the kernel) that can do them in bulk. That solves both perf issues (1 syscall, and no repeated round tripping through multiple address spaces), and the security issue (you no longer need to expose primitives to manipulate the remote process to every process that is allowed to spawn a new one).
The reality is fork() is pretty much exclusively used to launch new processes these days, outside a few specific cases. Today, it's a poor fit for that problem. And the answer is what Windows has been doing (and POSIX has now had) for a long time: explicitly launching processes by giving a handle/pathname to an executable like posix_spawn. That's the first solution, anyway; a better one would be more capability-oriented design where you have to supply a new address space with all its resources yourself.
This HotOS paper is a pretty good detailed coverage of the argument; I find it very convincing. If fork() went away, I honestly wouldn't miss it, I think. https://www.microsoft.com/en-us/research/uploads/prod/2019/0...
What does that mean?
> isn't thread safe
True but it's not meant to be. I'm not sure there are many if any valid use cases for forking and not immediately exec-ing and using threads together in the same application.
> insecure by default (inherits env/fds)
Inherits env and open file descriptors by design. It's pretty much always been understood that if you fork in most scenarios you immediately exec. You can set file to close on exec and set a new env if desired, and not do that if it's not.
> and it's also slow with all the state it must copy.
I thought it was mostly COW?
> And at a conceptual level it doesn't work in environments where the nature of a "process" and "address space" aren't synonymous.
Yeah valid argument. posix_spawn man page says:
> "The posix_spawn() and posix_spawnp() functions are used to create a new child process that executes a specified file. These functions were specified by POSIX to provide a standardized method of creating new processes on machines that lack the capability to support the fork(2) system call. These machines are generally small, embedded systems lacking MMU support.".
posix_spawn is POSIX and has existed along side fork since POSIX.2001. So your saying you want every application ever written to be automatically portable to systems the can't support fork, therefore get rid of fork entirely? I guess.
fork() plays havoc with threads. If you want to start a new process, specify a fresh process image.
> These services cannot expect to run faster from year to year with increasing CPU clock frequencies because the end of Dennard scaling circa 2004 implies that CPU clock frequencies are no longer increasing at the rate that was prevalent during the commoditization of Unix.
Definition:
> Dennard scaling, also known as MOSFET scaling, is a scaling law which states roughly that, as transistors get smaller, their power density stays constant, so that the power use stays in proportion with area; both voltage and current scale (downward) with length.[1][2] The law, originally formulated for MOSFETs, is based on a 1974 paper co-authored by Robert H. Dennard, after whom it is named.[3]
* https://en.wikipedia.org/wiki/Dennard_scaling
The article then mentions Moore's Law.
They can be running bare metal on top of a type 1 hypervisor, on top of a UNIX, Windows, mainframe, micro, or whatever.
Android and ChromeOS, which although based on the Linux kernel, don't expose a POSIX userspace to regular app developers as public API, only OEMs get to play with it, or placing the devices into developer mode like Crostini.
It doesn't matter at all on the XBox and Nintendo, but it does on PlayStation.
Its relevancy depends pretty much how deep one still is into classical UNIX like workflows, but like COBOL and Fortran it isn't going away anytime soon.
Unysis still sells Burroughs as well.
No, Android apps can contain native code that uses Linux syscalls.
"NDK C library"
https://developer.android.com/ndk/guides/stable_apis#c_libra...
"Improving Stability with Private C/C++ Symbol Restrictions in Android N"
https://android-developers.googleblog.com/2016/06/improving-...
"Android changes for NDK developers"
https://android-developers.googleblog.com/2016/06/android-ch...
Termux developers also think they could do whatever they felt like on Android, guess what, they can't.
The whole point behind POSIX was so people didn't need to constantly reinvent the wheel and glue things together to do simple things on different platforms. Today if you need to do anything in the cloud, you must "integrate" the pieces, which means to take out scissors and glue and toothpicks and a text editor and literally arts-and-crafts the pieces together. Want to add something to your system? More arts and crafts! Rather than waste all that time, if everything just used a common interface, everything would just work together, the way it (mostly) does with POSIX.
This is bad for the companies that make all their money off proprietary/incompatible interfaces - lock-in devices that are advertised as a plus because they can be "integrated" with enough time and money. But it's better for our productivity and the advancement of technology if we didn't have to integrate at all. Just run the same thing everywhere.
And yet the most pleasant development environment for producing the latter is going to be the former.
> Virtual memory is considered to be a fundamental operating system abstraction, but current hardware and application trends are challenging its core assumptions.
What are specifically challenging the vm assumptions?
I suspect he may also be thinking of the prevalence of one virtual machine:one app in the cloud, where you can argue (and the Seastar guys do) that the process abstraction just slows you down because the unit of isolation is the VM anyway.
If you mean the underlying data structures, that's basically what modern filesystems are. XFS, APFS, BTRFS etc are all copy-on-write file systems that use git-like structures underneath. In the same way that git branches are "instant" because no data is actually copied, so too can these file systems clone files or whole trees instantly without copying any real data. You can easily "branch" a directory tree, rsync the results to a remote machine, etc.
I get what you're saying about modern filesystems, and I agree. I guess from that POV I'm saying we could stand to remove some of the layers of abstraction?
At any rate you might be interested in a few different projects:
1. BlueStore: https://ceph.io/en/news/blog/2017/new-luminous-bluestore/
2. The DAT or IPFS protocols, which are based on the idea of immutable logs storing file data, identified by hashes, with public keys and signatures to handle mutability.
It would also be nice to support filesystem transactions, somewhat analogous to Git commits. POSIX file APIs make it difficult to avoid race conditions in the best circumstances, and extremely difficult if there’s a security boundary involved. You can never check something about a path (e.g. “is this a symlink?”) and then rely on that thing being true, because it could have been concurrently modified between the check and whatever you do next. So you have to rely on the limited atomic semantics available - for example, instead of explicitly checking for a symlink, you can use O_NOFOLLOW - but those are not always sufficient, depending on what you’re trying to do. It shouldn’t be this way. I should be able to take a frozen snapshot of the filesystem, inspect it to my heart’s content, make the changes I want, and finally atomically commit or abort.
Regarding copy-on-write clones, can any of those filesystems actually clone arbitrary directories? In APFS’s case, you can make copy-on-write clones of files, and you can make copy-on-write snapshots of entire volumes, but you can’t just take some random directory and make a copy-on-write clone of it (without individually cloning all the files under the directory). I believe the same limitation exists for some or all of the modern Linux filesystems.
Agreed about fs transactions. Not sure about cloning directory trees. I thought btrfs could do it but I never used that feature. You might well be right.
https://storagedeveloper.org/events/sdc-2022/agenda/2022-09-...
"Symbolic links Considered Harmful"
Might be relevant to readers :-).