The operating system: should there be one? (2013) [pdf]
citeseerx.ist.psu.edu
citeseerx.ist.psu.edu
I think modern OS stacks are in a similar boat, at least Windows and macOS. True low level control is kind of a pain, but the OS libraries are not particularly pleasant or high level enough either. So largely things like Electron or Qt are acting as the defacto OS anyway to varying degrees of success but undeniable levels of resource waste. I wonder if modern languages and tooling and such are advanced enough that the OS should become more primitive and minimal. In a way it feels like we're moving there anyway with web tech becoming the defacto user space and toolkit. That's fine, but god if it wasnt all built on 50 layer of cruft our machines would be screaming fast.
Smalltalk, if you get a chance to use it, really does act like its own OS in so many ways. It's great as a programmers OS really, although it doesn't really hit the right notes for average use. Still though, if someone had the urge to build it I could easily see a smalltalk based userland built on like a linux kernel (with no gnu user space) being a totally viable thing.
The existing issues aren't technical, rather political, like the WinRT crusade that ended up bombing and now we have plenty of GUI toolkits to chose from on Windows.
Swift, Java, Kotlin and the .NET languages are more than high level enough.
CDE? That hasn't existed for decades.
macOS, although a UNIX, follows the same ideolagy as NeXTSTEP, where UNIX compatibility was a means to bring software into the platform, and that was about it, GUI software was to be fully taken advantage of Objective-C Frameworks.
(Hell, even UNIX itself doesn't exist anymore. Linux and FreeBSD is its own thing now.)
As proven by https://news.ycombinator.com/item?id=28437173 making to the first page today.
You missed my praises for Apple and Google platforms.
[X] http://jdebp.info/Softwares/nosh/user-vt-screenshots.html
That's in no way a "classic Unix desktop". This is hipster nostalgia, like those indie "retro" pixel art videogames.
Yes. "CDE" itself stands for "Common Desktop Environment". (Wikipedia says it was "part of the part of the UNIX 98 Workstation Product Standard", and if you remember the 1990's like I do, it was, indeed, supposed to be the "standard" desktop for Unixes.)
P.S. Linux isn't a Unix.
https://sourceforge.net/projects/cdesktopenv/ https://sourceforge.net/p/cdesktopenv/activity/ https://sourceforge.net/p/cdesktopenv/code/ci/master/log/
Instead, developers rely on toolkits such as Qt or GTK to ensure compatibility, which are usually either bloated or ugly.
IIRC the fancier toolkits (Qt, GTK) do widget accell with either mandatory blitting and/or 3d (OpenGL/DirectX) (Other frameworks like WPF and Cocoa are notorious for this too).
There are, or used to be, frameworks that were not gpu dependent.
I've never hear nor seen these massive drops you claim to, so either you are running 20 year old hardware or your graphics aren't on, or you have some other hardware issue...
When I said "I don't understand the technical reasons behind it", I didn't mean "I don't understand how to abstract OS-native APIs". I meant "I don't understand how a team of intelligent people managed to fuck things up this badly".
Get your favourite Qt program, resize the window like a madman and watch your CPU usage launch into the stratosphere.
You’d probably also want some form of malloc(), but that’s completely optional. There’s nothing in Objective C that says you have to allocate memory dynamically, or that you have to do it with malloc.
ObjC runtime has grown somewhat to include more features, but you don’t need all those runtime features if you want to run your code on a microcontroller. Just like you don’t need glibc if you want to run C. There is more than one runtime for Objective C you can choose, just like there is more than one runtime for C.
You can have all the 'simple' calls you like but if you need to malloc half a gig of ram just to get started...that is heavy and bloated.
Also you say 'architecture' but unless you are talking a battery hungry cellphone you are probably talking 16 or 32 bit proc not 64, which means potentially massive increase in the size of code to be generated since you lose certain instruction sets. I don't know realistically what Obj C uses in the osx cpu architecture but I'm almost certain it's not going to be as simple as just retargeting your compiler...
Not saying it can't be done but take even the new arm laptops from Apple - that's a significant hardware investment on top of a bunch of software tricks, not just a casual retargetting that can be portably moved to other low power systems.
Objective C is in mainline GCC. You retarget your compiler and port the runtime. As far as I know, any target with GCC support can run Objective C, given a runtime. The important parts of the runtime require kilobytes, not megabytes, of space.
Remember that Objective C was originally a very thin layer on top of C. You take all the [method calls] and replace them with calls to objc_msgSend(). The overhead is fairly small— your object instances need the “isa” pointer, which is basically just a pointer to a vtable. The objc_msgSend() function reads the vtable and forwards method calls to the correct implementation.
My experience is that Objective C binaries typically have a much smaller footprint than C++ binaries. That’s obviously not some kind of rule, but it reflects idiomatic usage of Objective C. In C++ you'd use a std::vector<T> which is a templated class, which gets instantiated for every different T you use in your program. In Objective C, you’d use NSArray, which is monomorphic.
This all should be completely unsurprising—since Objective C first appeared in the 1980s, it’s no surprise that it doesn’t need much memory.
It's definitely a lot more than the NeXT days. GCC got fast enumeration, exceptions, garbage collection (if you really want it), synthesized properties, etc. These are the "Objective C 2.0" features which got released in 2007, back before Apple was shipping Clang.
GCC doesn't have ARC, array / dictionary literals, or other new features. But it's definitely a lot more than Objective C from the NeXT days. These are "modern" Objective C. They're also basically just sugar for the appropriate calls to +[NSArray arrayWithObjects:...] or -[retain] / -[release] etc.
Debating whether some language feature can exist on an mcu is somewhat unrelated - the point was about code size bloat not whether you can fake one instruction set with another...
You can fake e.g. multiply with a for loop and an add instruction (1 or 2 cycles) but that will run orders of magnitude slower than a 2 or 3 clock cycle multiply instruction...
So the point about osx arch is not really addressed, if you have runtime that cannot be easily ported or osx cpu arch which produces prohibitive instruction sets it doesn't matter if gcc can target an old version of objective C, it is, as stated, not that simple.
I don't think the newer Windows UI-Frameworks are loved that much and UWP is not really convincing for desktop applications. Just too much hassle for too little gain and the threat of vendor lock-in.
But yeah, having ramped down the team while beting the farm into WinRT wasn't the best idea.
Still, just like VB 6 in Windows 10, WPF will be around for decades to come and it isn't like there are revolutionary UI concepts to implement.
I used to be a desktop dev but after the mess with Winforms, WinRT, WPF, Silverlight, UWP and WinUI I would never bet again on an MS GUI framework. Most likely it will be abandoned soon and put into maintenance mode.
I still don’t understand why they constantly crank out new frameworks that do basically the same instead of evolving an existing one like WPF. The cost of enveloping, testing and documenting these frameworks must be enormous.
There were three big issues although I can only remember two, which were WPF and WinFS. I said flat out WinFS isn't going to make the cut, the third feature wouldn't, and even if WPF did make it into the final build, Mozilla doesn't have the resources or desire to move Mozilla to WPF, especially since mozilla is a cross platform app. "Yes, we know, but utilizing WPF will make it easier to develop on Windows!" I pointed out while that's good for MS, it provides no benefit to Mozilla's non-windows users, and Mozilla would never do it. And WinFS isn't working at all, so why would they spend even a moment trying to figure out how the new FS would benefit an application?
I think the third leg was Palladium. "It'll make online banking so secure!" I remember commenting that with all the flack it was getting, it'll have no buy in from anyone else, and flop.
I don't think compulsive "tell it how it is" people are good for evangelist roles. :D
I come from a background as a EEE mainly writing embedded code before somehow stumbling into frontend code (in win32, Cocoa and Qt) at my old job.
From my experience, OS-Level APIs definitely have more in common with Electron or Qt than they do "bare metal". To make the top left pixel red on a bare metal system or RTOS, you simply write `framebuffer[0] = 0x00ff0000`.
For win32 or Cocoa, you need to create a window, alter its flags to ensure it's borderless, change its size and posistion to 1x1 and [0,0] (or [0, screen_height] on the Mac), strip away the drop shadow, respond to regular paint events and pray like hell that nothing else draws over the top of it.
>True low level control is kind of a pain
It depends on what you're trying to achieve. If you're trying to get something highly responsive that looks exactly like the UI spec, most libraries tend to make this job harder when the abstractions leak and you're forced to rely on bizarre hacks that make no sense and feel genuinely demoralising to implement.
Dear ImGui, which exposes far more of the low-level internals than any other API, is an absolute joy to build GUIs in and significantly easier than something which "holds your hand" at every step.
As much as I struggle with Qt's APIs (aging janky Widgets, half-baked QML on the desktop, the item models are an impenetrable byzantine enterprisey API that's simultaneously too open-ended, not flexible enough since it imposes its add/remove/move schema on your model, and too buggy), building a GUI from the ground up is not going to give you internationalization or accessibility, and Qt is.
Vanilla ImGui doesn't, but I both wrapped my head around and then rewrote the text renderer in all of 2 days as a junior dev.
This version could support arbitrarily many UTF-8 codepoints with kerning and all and had a negligible performance hit on the machines I tested on. I never upstreamed it (left the company before the product that was using it shipped, actually), but it wouldn't be too hard to reproduce.
That's what I love about exposed, lower-level frameworks like Dear ImGui.
We didn't aim for accessibility, but I'm sure that would have also been semi-trivial for a dedicated dev who understands what's required.
The other major thing missing in relation to that is input method support.
The problem with those type of APIs and accessibility is that implementing screen reader support basically means making it retained mode in some capacity, because accessibility APIs expect you to export a tree of objects and push state updates.
Looks like theirs is far more advanced than mine (mine would just generate a texture per string and cache it, the main problem it solved was being able to display non-ASCII filenames). I don't think there's much I could contribute to them at all.
>accessibility APIs expect you to export a tree of objects and push state updates
Is this an inherent limitation with the tech itself, or just an arbitrary API limitation?
Of course this does not apply if all you have is a screen connected via SPI to an MCU's GPIO pins. But you won't expect much graphical sophistication from such a device anyway.
I could imagine if you had decades of experience using retained-mode, though, then you'd find that way of thinking more natural.
Game UIs are (from what I understand) written almost exclusively ImGui-style, with every single widget being fed updated state data every single frame as part of the game loop.
Yet, for whatever reason, even crazy-complex UIs like SupCom's seem to have no issue rendering in realtime with an entire 3D game running in the background, while simply resizing the window I'm typing into right now (a single textedit on a static HTML page) creates visible lag and chews up full CPU on a 2015 Macbook.
That's not to say retained-mode GUIs can't be performant (embedded systems rely on retained-mode to have reasonable performance), but on a modern system it seems like the loss in performance caused by inefficiently synchronising state in retained-mode GUIs is several orders of magnitude higher than the losses caused by pushing a couple of hundred textures every frame in an ImGui context.
- Wrapping text labels
- Widgets that are similar to CSS flexbox where the size of elements depends on other elements
- List/tree widgets or text editor widgets with large datasets in them
And then you will notice performance tanks. Think of trying to build a text editor that way, if you opened a 5MB text file, you would have to rescan the whole 5MB and compute reflow for it every time.
I do often reflect on how fast game UIs draw, and compare that to the headaches we have in the UI of a digital audio workstation. It generally seems almost inconceivable how we could be so slow in comparison. But then I think about all the ways the user has to interact with the actual UI (not the backend data model), and it starts to seem like a different sort of programming model entirely.
Overall, I really like Dear ImGUI and I love how easy it is to expose data to a UI through it. Its much more pleasant to use than a traditional retained mode widgets API. But Dear ImGUI isn't a good general purpose UI (nor was it designed to be one), although I could see the immediate mode API work well for one. I started making a complex editor for a toy engine in it and while it worked ok, the reasons above made me cut it back to just some basic in-engine stuff and the rest is exposed over an optional embedded web server module and a react app as the editor. Its only a toy, but you would use the in-engine ImGUI editor to see the scene rendered in-engine, move objects around etc, but you would use the react app to edit object properties, add assets and behaviours and whatnot. It made it much easier to create rich editors for things like behavior trees, node graphs, etc. Although since its just a toy, I didn't implement a lot of this stuff yet and haven't had time lately to get back to it.
The desktop GUI is just one subsystem in a modern OS, and you don't have to use it if you don't want to. If you do use it then there are tradeoffs, of course, but that's always true of anything you use.
Never cheated in Minesweeper, I presume?
Really? In my brief experience it was more like sending a bunch of commands over SPI/I2C/UART to some LCD module. Which isn't thaaaat different from how an OS driver communicates with a GPU, only the commands are far, far more complex, so using an OS-level API avoids having to do it by hand.
https://github.com/torvalds/linux/blob/master/arch/x86/entry...
Writing into a framebuffer is arguably as much of an abstraction these days as many of these toolkits. In the modern world of GPUs and display controllers a framebuffer itself is already N levels away from the "physical" display hardware. We're not in VGA land anymore. Some aspects of the OS or the OS's interaction with the display driver may present something like a linear framebuffer to you, but underneath that is a world of GPU textures and buffers and display controller abstractions.
When I think of OS-level APIs I think of ioctl, mmap, file descriptors and sockets. I guess that's my Linux bias showing?
I know right now Microsoft keeps making like 50000 new UI frameworks and part of me is like, just knock it the fuck off, give me some style guidelines and some super-efficient very low level no-frills API's and let me just build apps off a library like Qt that abstracts it.
Wait. In that case, library developers or OSS projects can fill the gap. What am I missing here? QT comes to mind. A high level library for GUI programming. Also, the modern .NET stack.
I always thought the popularity of Electron is because GUI programming is insanely easy on it, and if you have apps that are heavily data dependent, you go with it because the alternatives (Native apps, WPF or Winform apps) do not provide any value addition.
I would say that if you just wanted to spit out a Windows application, and you already knew the technologies, WPF or Winforms would probably be better (despite the fact that WinUI is the new hot stuff).
Smalltalk can provide a storage abstraction, but to interoperate with other things it needs to be able to save data in a way that can be loaded back into your Java program.
OSes can provide you with the desired abstraction level.
You could stop at the block layer, which many DBs can take advantage of, but that won't help your Java and Smalltalk program work together easily. A filesystem is a common abstraction which should provided by someone.
Your example about the graphics stack is no different, the problem is that various components of the system are not "shared" in the same way all the time. The actual low-level abstraction of the GPU being provided to you by the kernel has more complexity today than what it had years ago in order to allow virtualization and compartmentalization of graphics memory. This wasn't needed some years ago, as it was assumed X11 was the only program to ever going to access the framebuffer, and would allow any single program using GL to trash the entire GPU.
What has been simplified is the user-level facing code, so that you can more precisely control allocation of the resources. Akin to the page-level memory allocation support we always had. This was done to allow /some/ programs (3d engines) to push the hardware to the limits (I'm relatively sad the GL1.x is "deprecated" - I still consider it to be an excellent API for prototyping and what would be 90%+ of applications).
Consider audio instead. OSS always provided pretty much all the hardware features, but allow no resource sharing. Every single evolution on that front has been an increase in complexity to allow for resource sharing. And it's interesting because the effort went in and out of the kernel a couple of times. Describing the evolution of OSS/ALSA in linux/freebsd would be too long, however we now have very fat daemons sitting in front of the HW just for resource sharing.
None of the API changes were required for resource sharing though so that complexity could (and should) have been contained to the implementation and those applications that actually need new features.
This is not true. Early evolution of ALSA included lots of work on handling high channel count, memory-mapped devices (and their opposite, a few devices around in about 2000 that require active CPU involvement in data transfer). At some point, more evolution was required to deal with asynchronous interfaces like the USB ones that are now ubiquitous. The Intel HDA "specification" created a huge amount of work to model the topologically indistinct hardware mixers that it allowed. It remains the case that ALSA provides no facilities for resource sharing that are used by more than a handful of obstinate folk, so there has been essentially no evolution within ALSA itself toward that goal (see next para).
> And it's interesting because the effort went in and out of the kernel a couple of times.
Once OSS was dropped, the only resource sharing left on the kernel side in ALSA was the dmix layer. That continues to exist today, but has never worked reliably, which is part of the reason why PulseAudio came into existence.
> Describing the evolution of OSS/ALSA in linux/freebsd would be too long, however we now have very fat daemons sitting in front of the HW just for resource sharing.
I object to JACK being called "a very fat daemon". It's lean and mean and fairly clean (even the version I didn't write). Perhaps you mean PulseAudio, but that does much, much more than just "resource sharing".
As a side-note, Apple did move resource sharing out of the kernel. At some point that was all kernel-side, and at some point, coreaudiod showed up as a user-space daemon that was eventually clearly doing the same thing.
That's the natural (and arguable optimal) state of things: lower levels and higher levels.
High-level developers use high-level things, low-level developers use low-level things.
Every once in awhile someone gets the bright idea to combine all the layers into one, but finds out that in the real world that just isn't practical.
But we get developers pulling apart and adding layers over and over, driven by a love for "simplicity." Moving forward, or in circles? Hard to say, but probably forward.
For example BIOS used to be a driver layer back in the day, providing simple API to underlying OS. This concept should be expanded. I don't see no reason for every OS to rewrite those drivers again and again. There should be some standard for common device APIs, like disks, keyboards, mouses, GPU and so on. I should be able to buy an Nvidia GPU, download and install UEFI driver and it should provide Vulkan API for any underlying operating system, Windows, Linux, OpenBSD, ReactOS.
Another example is file systems. All major operating systems are very similar when it comes to file systems, at least from the user PoV. Yes, there are Windows ACLs, Linux permissions, but it should be possible to abstract those away enough. So Windows could install its latest Windows 11 NTFS driver into UEFI and Linux should be able to just use it to access NTFS safely, without any gigantic efforts to reverse-engineer and reimplement everything from scratch.
At one point I thought about virtualization being an answer to those demands. These days you can write completely virtualized operating system which needs to implement virtblk, virtfs, virtio-* drivers and run it on any host operating system with perfect driver support from manufacturer. GPU is not there, but I don't think that it's impossible to implement. Of course running a full-blown ad-ridden Windows 11 just as a hypervisor seems like not the most elegant solution.
Unification will not play nicely with new progressive features like BTRFS, though. That's something to think about as well.
Thing is, most devices you actually don’t need special drivers because they are so standardized. Mice and keyboards are USB HID. There are a handful of ways you typically interact with storage, like using SCSI. Surprisingly, a lot of storage devices are actually SCSI, in some form or another.
One of the big problems is that drivers are going to closely interact with other operating system services, like memory allocators or the scheduler. The interface has gotten a lot more complicated over the years. And the semantics that operating systems present to the user are sometimes different in significant ways. Back in the BIOS days, what you could do through the BIOS was rapidly falling behind what you could do with a proper driver.
To pick on the filesystem example, there are some fairly major differences between Windows and Linux/macOS/BSD filesystems APIs from the developer perspective. Windows has mandatory locks and files are identified by name. On Linux/macOS/BSD, there's more of a separation between a file and its name—files do not even need to have names at all, and locks are advisory. For this and other reasons, there are some boundaries between files you use in WSL and files you use in Windows.
I’m just very skeptical here that you could introduce another layer between devices and the OS, that would be thinner and more direct than using a hypervisor, more flexible than just using a standard protocol (like USB HID) which your OS already supports, and work with multiple operating systems.
For example I read on LWN that hardware vendors are starting to look at Linux virtio for hardware agnostic protocols.
It is interesting that abstraction always comes a full Circle.
Then, we'd have reference implementations for every single one of those, that would give examples of how to access this standardized functionality, possibly with abstractions over those. So on one end you'd have something like a wi-fi-networking-huawei library and on the other you'd have wi-fi-networking-generic library that could interface with the former and access its common functionality. Of course, as for how this could be written, and what the situation would be in regards to the C standard library or any other primitives for other sufficiently low level languages, i have no idea.
Then, you'd have a kernel module or a microkernel, that would contain the code to interface with the hardware and add all of the controls and permission management functionality, as other bits to integrate this solution with the OS. If one would go with the GNU Hurd approach, then integrating that within OS distros would suddenly get even more simple, except that Hurd seemed to fail at the actual implementation stage.
And finally, you'd have the OSes, which could implement code to interface with these modules, microkernels or add any number of necessary shims inbetween to get everything working together and provide the higher level OS functionality that's needed to use these devices within the OS context.
Of course, i'm probably oversimplifying some things, skipping a few steps here and there, but i'd chalk that up to the current approaches to using hardware in software being overcomplicated beyond need or belief, rather than me mostly being a business software developer, as opposed to systems software developer or hardware engineer. In short, there are abstractions in place already, i'm just not sure those are the right ones, but rather what was historically needed to get things working. So where's the big rewrite and learning from our past mistakes?
To me, it just feels like what we need is a set of meaningful abstractions and standardization, and not just in regards to making sure that there are security mechanisms in place for booting so the devices on which Windows runs can be limited to create more e-waste (snark) and motivate consumerism, but also to actually be able to use Wi-Fi in all hardware in 2021 in both Windows and Linux distros (not snark, literally had to compile custom drivers for my no name netbook).
If you see the operating system as the task-schedular + IPC clearinghouse then the system is simple, well defined and it can be expected to perform in a deterministic and verifiably correct way (assuming working hardware).
If you define the operating system as 'everything and the kitchen sink' including media players, browsers and so on then there is no way to make any guarantees and determinism is going to be impossible to achieve.
Personally I prefer the leaner-and-meaner operating systems that have a larger ring of userland software built on top of them to create as small as possible a core on which the actual end-user applications are run.
Those systems tend to find good middle ground between reliability, speed and security because the amount of surface area exposed is going to be the minimum at every level.
Something like Windows is on one end of that scale, something like MacOS is somewhere in the middle. Linux can be stripped (or more accurately: it could be stripped) to something a lot slimmer than the default distros, and something like IRIX (no longer used in production as far as I know) was roughly where MacOS sat (custom hardware, custom OS tied closely to that hardware).
An ideal OS that limits itself to scheduling and IPC is currently not on the market for desktop usage, and the arrival of the mobile device market has blurred many of the lines that we thought were drawn pretty clearly in the past.
The 'lets move everything into the kernel' philosophy is as far as I'm concerned fatally broken and likely things will have to get a lot worse before they will finally get better, and it would not surprise me if everything that we do today will be discarded once that happens.
I've wondered the same thing about databases. Just like an OS, a database has to manage data on disk and figure out when to cache it. The files & folders metaphor is arguably a relic of pre-computer times--instead of searching through folders, it is easier to search and sort based on particular attributes, something that databases are well-suited to. I think it'd be interesting to start with a database and try to build an OS around it, see how far you could go.
That is, unless there exists some kind of tree structured database that I don't know about ...
Many years ago Microsoft tried to implement some kind of database oriented file system (it was a big dream of Bill Gates IIRC and probably meant for Longhorn), but this project never became part of an official Windows release AFAIK. It'd be interesting to know why Microsoft abandoned this initiative.
There were technical issues. They simply never managed to get the system performant enough to be use as a FS replacement. There also were tons of issues with the concept itself, like proper tooling for visualising the stored objects to the user and a compatibility layer for legacy software.
It seems as if parts of it was merged into MS SQL server products [0].
People like doing that until they have sufficiently essential complexity that it breaks down, then they hate it with passion.
Then when they decide they actually need tags, you either get to be a hero, or to goof off for a couple weeks, depending on your situation :-)
Nesting works the same way. "OK we said we needed N levels but actually we need arbitrary levels of nesting, and also we don't realize it but what we're asking for can introduce cycles to the structure". Every. Damn. Time.
It's a tyranny of hierarchical data-structures and ontologies. Always breaks down. Use relational models, graphs, tags, sets, uuids/global identifiers etc. Trees are an optimization strategy/snapshot and almost never a model for the real world.
Believe it or not, hierarchical databases were actually quite popular once upon a time, mostly in IBM mainframe land as I understand it. Today the Windows registry is a hierarchical database in common use.
The Be operating system (BeOS) uses BFS as its native file
system. BFS is a modern 64-bit journaled file system. BFS
also supports extended file attributes (name/value pairs)
and can index the extended attributes, which allows it to
offer a query interface for locating files in addition to
the normal name-based hierarchical interface. The
attribute, indexing, and query features of BFS set it apart
from other file systems and make it an interesting example
to discuss.
https://docs.oracle.com/en/database/oracle/oracle-database/2...
Then use APEX for Web applications, that make use of stored procedures,
Other enterprise RDMS have similar offerings.
"Automatic Storage Management"
https://docs.oracle.com/en/database/oracle/oracle-database/2...
Allows mapping volumes to be directly managed by the database.
Neither hierarchical organization nor tags nor databases work. You need a mix of everything, and you need to make it nice enough for the user to be able to select the best option at the right time. Think about how you think and access memories. There are a ton of ways you do it, not only hierarchically or "table-based".
This could also be likened to the problem of categorization that meta-rationalists like to talk about so much. Basically, that (at least in human terms) no categorization works under all possible contexts.
We are not gods, so it should be obvious that there's no possible organization of knowledge that makes it all perfectly available. It can't be frictionless, instantaneous and complete at the same time. Then, it should be obvious that if we want to make access to information as effective as possible, multiple methods need to be combined.
In fact, a lot of the technology that has been most "revolutionary", has been technology that improved the way we access certain types of information. Graphical interfaces, databases, spreadsheets, hyperlinks, google search, etc. We have also used calendars, lists, schedules, diagrams, button panels, etc., long before computers were a reality.
There are still a few steps left. As we attempt to tackle even more and more complex systems, we are pushed to look for new ways to manage all that.
>> One has to be able to “bend” concepts, when it is appropriate. Nothing should be absolutely rigid. On the other hand, things shouldn’t be so wishy-washy that nothing has any meaning at all, either. The trick is knowing when and how to slip one concept into another.
And reflect again at this point about what you have read:
>> The point of these examples is that what counts as an “object” depends on the context.
...while thinking explicitly about the categorization problem. In computer science and AI, in particular, this problem might appear even more clearly, specially in symbolic AI. If you try to create a symbolic AI system, or if you are an experienced programmer and have a solid grasp of the limitations of abstraction, it all floats around this main issue of nebulosity / categorization / methodology in rationalism. You might also be reminded of the aphorism "all models are wrong".
Hope this gives you a starting point to keep exploring more deeply if you want to.
This can lead to some spectacular results for things like TCP. Having the ability to serialize a TCP connection and sending it over the network means you're able to transfer a TCP connection from one host to another, even with SSL on top.
Now, this can be done with a POSIX kernel as well, as long as you move the networking from the kernel into the application. But if you're doing this you're eroding away the benefits of the kernel, basically turning your OS into a bootloader. Same thing with scheduling.
Another benefit is that, at least on compiled languages, code that isn't used isn't there. So the linker can strip away everything that is never called.
This still feels like utopia. The average development environment is pretty complex and ripping out and replacing all the layers below your language runtime is too much effort.
Of course there are lots of other overlaps between software and postmodernism in other culture, and that was the wider observation of the Noble & Biddle essay... true that it is a bit oblique. You might enjoy this a bit better: https://www.se-radio.net/2006/11/episode-38-interview-james-... -- it's now slightly dated, e.g. how "programming by Google / StackOverflow" was a much newer idea then than it is now.
I don't want to converge towards a single OS, but welcome standards and bridges between them.
Similarly to how you don’t really get competitors to eucaryotes and prokaryotes, the initial investment (device drivers) to create an OS (organism) that is actually usable is so high that one basically can’t overcome it when others are available.
http://www.softwarepreservation.org/projects/LISP/interlisp_...
Then follow up with "The Interlisp Programming Environment"
https://www.computer.org/csdl/magazine/co/1981/04/01667317/1...
"Graphical Programming in Interlisp-D"
https://www.youtube.com/watch?v=wlN0hHLZL8c
"Eric Bier Demonstrates Cedar"
https://www.youtube.com/watch?v=z_dt7NG38V4
Cedar was the very first graphical workstation for something like .NET and Java would later kind of try to become.
Because
&
On other hand - define OS. Isn't that Smalltalk environment an OS if it runs bare metal?
Yes I'm saying the Unix file system is a worse mongodb. Please at least give me a postgres or something.
Everything in 'special' mounted filesystems like procfs, sys, cgroupsv2, etc as a start. These all use implicit schemas of particularly named files with particular contents that you have to learn about out-of-band to just to do anything.
Requirement 1: Everything has an inspectable schema.
All the configuration files with their own home-grown format that you can only validate by restarting the program that relies on it and hoping that it doesn't crash your system is for the birds.
Requirement 2: Files that must have a particular format must also have a well-defined Type that can be validated to be correct in a uniform way outside the program that uses it.
Isn't that exactly how traditional FS work anyway? Even using MIME types only gives you an optional, non-binding hint. In the end you need to parse and interpret the content "manually" (well, usually using libraries).
> parse everything manually just to interpret it: not so great.
Except the first sentence, the first paragraph describes current filesystem design.
Maybe I've completely misunderstood your proposition, though.
Now, that doesn't stop MyProprietaryApp from declaring that the type MyProprietaryFile consists of a big bucket of bytes which it then interprets how it likes, but that's inevitable and not worth worrying about.
The point is to enable defining directory and file schemas for my OS and applications that I can control so that interaction and discovery within these systems isn't so baroque.
None that immediately spring to my mind. There are operating systems that incorporate programming languages as their default user interface (like UNIX that drops into a Borne shell and LISP machines) and there are computers that have programming languages baked into their firmware (like the 70s and 80s micro computers did with BASIC). But in neither instance is the language itself the operating system.
The micro computer instance is definitely nuanced though. Some do argue that Microsoft BASIC et al were an operating system. Some don't. I'm in the latter category because I think "BIOS" more closely describes the function of the software on those old micro computers.
- Pilot was an OS that was essentially bare metal Mesa. It was written in Mesa, only runs Mesa programs, and used Mesa language features for OS capabilities like safety: https://courses.cs.washington.edu/courses/cse550/20au/papers...
- Cedar was both a PL and an OS: https://www.ics.uci.edu/~andre/ics228s2006/swinehartzellwege...
- The Oberon OS and Oberon PL are highly coupled https://people.inf.ethz.ch/wirth/ProjectOberon1992.pdf
- Here's a Scheme OS called MrEd: https://www2.ccs.neu.edu/racket/pubs/icfp99-ffkf.pdf
What happened was some academics figured out it was possible to make virtual machines on the 370 and that you could run a few different operating systems at the same time. If you were doing software dev on the 370, for instance, you would start up a VM running VM/CMS which was a single-user OS a lot like CP/M or MS-DOS, work there, possibly using services offered by another OS in another VM by some kind of message passing.
This turned out to be the "master plan" in the long term, since they could have a few different mainframe OS, even some applications that run on "bare metal", but have it all run on one machine or one cluster of machines and be managed under the same pane of glass.
Eventually, this lesson will spread. I think about 5 more years of chaos until people finally catch on.
Also there are many x86 and other systems where people use VMWare or something like that for "server consolidation" without going all the way out to the cloud.
The difference today though is that the guest operating system today is a "full-service" operating system such as Linux or Windows. In many of these cases the partitioning of the service could be accomplished through user id's, processes, sessions, containers, and similar facilities offered by the guest OS.
But the central idea that the server system can only run one program makes a lot of sense to me. Obviously it won't work for a desktop, but then the whole idea that desktops and servers run the same OS doesn't make much sense to me.
Say I have an web API written in Go. For some reason I need to do some openssl stuff before I start up my main program. In a container I can just wrap my program in a little shell script which generates/signs a cert before starting the API.
Such a simple task. It is gonna be a lot harder with a Go Unikernel as I could have to bake this functionality into the application itself.
This Swiss Army knife that is Unix often has it's uses.
In any case, in you have a web API written in Java then you can connect advanced monitoring tools like JDK Flight Recorder and directly affect the system via JMX beans.
So it is more a matter of tooling maturity than anything else.
But the reduction in attack surface of not having all those Swiss Army blades hanging about - and not having to configure them just right so I don't accidentally damage stuff - is a real benefit.
In those cases 99% of the time you really don't care what runs underneath the language runtime.
So upgrading the capabilities and creating new common abstractions in an OS is not that hard. But almost impossible to get the majority to go along with it.
Browsers demonstrate an additive way of getting a lot of that. In that we now have essentially two highly compatible widely deployed VMs -- Firefox and Chrome. Which can be programmed in a number of languages that compile to JavaScript or web assembly and share a rich API.
Web assembly modules seem relevant.
https://github.com/WebAssembly/proposal-type-imports/blob/ma...
On the other hand, if you're even thinking about touching the internet, or any other computer, you need to have every single task limited ahead of time to a specified set of resources, like mainframes do. Doing anything less is foolish, and I certainly wouldn't build a civilization on such folly.
Yes, in an era of mobile code, persistent internet, you need an operating system that doesn't give any default permissions to any executing task, ever. It must also safely and securely multiplex access to the hardware resources, all of the hardware resources, in all of the subsystems in the SoC, etc.
There are no Operating Systems, as of yet. I weep for the future.
It would be, if I could get it to run my PC. Same with Genode. Of course at this point nobody expects HURD to come back to life.
If you have more than one program, they need shared data and access procedures, like the “user” and “the hardware”. Who manages them: The OS?
Maybe you want to share a bit more like some common higher level functionality. That becomes part of the OS, de jure (resident kernel) or at least de facto (loadable kernel modules).
Some game on a MS-DOS floppy disk that boots itself from the boot sector and uses BIOS calls has an operating system.
The set of requirements which describe the host environment (the "virtual machine" in the classic sense) that forms the backdrop against with an application is developed is the operating system.
The only way you don't have an operating system is when the application itself does everything it needs with regard to the outside world by itself poking at I/O registers, and handling every interrupt and so on. Anything independently organized for that purpose, given something resembling an API, is an operating system.
There is, or used to be, a sensible version of "there should not be an operating system" I first heard some decades ago. Namely this: "there should not have be an operating system with a hardware-protected user/kernel boundary". The idea is that there doesn't have to be kernel which the hardware conceals using privilege levels and fences. The idea is supported by the notion that all security can be handled at the software level. The system, as such, is somehow verified to be correct. The only way to execute code is via compiling it via the system's trusted compiler, which generates only safe instruction sequences. Those sequences can call functions in the system, and those functions sandbox what can be done. All notion of trust and privilege is software-level without any privileged instructions having to fall down a trap into a privileged kernel which meticulously validates arguments.
(All that rhetoric needs to be somehow adjusted for 2021, which finds us jaded by knowledge of side channel attacks, cache attacks and whatnot.)
Browsers have a set of APIs that are decided on by a standards committee, and you have to implement those standards if you want your browser to be compliant. As an application developer, all I need to do is check a page to see how widely a standard has been adopted and then use it - I can be sure that my code can ship to (nearly) every device and behave uniformly, with minimal tweaking.
If the operating system had gone that route, we wouldn't have had to build more layers on top of it to abstract away the differences.
I agree that a filesystem as we know it in unix-like systems is a set of objects (directories) and files.
Would it not be preferable to retain this simple interface and instead conceive of interaction between programs and threads not as a stream of distinct objects in memory, but rather as a data tree within each program's control which can be exposed as input or output for other programs to map against and write into / consume?
Complexity, particularly NOT-self-documenting complexity, becomes a nightmare.
Imagine instead that each program controls a special area under /proc/(pid)/ which behave like tmpfs to that program: named interfaces are subdirectories, messages are files.
Reflection of supported interfaces becomes easy. It integrates with existing software, and it's self documenting for both software and users.
Complexity can be opt-in rather than the only way to operate.
That's API. Users will depend on it. So now you either have to freeze your code or break users flows.
This is basically how Plan 9 works (or so I’ve heard).
- Claude Shannon, A Mathematical Theory of Communication
Everything is inherently a stream of bytes. You can use programs that interpret them as typed data or newline or null-delimited arrays all you like.
Programming languages create idioms and standard ways to do things. We judge those for their clarity and quality. On that front UNIXs "you get a stream if bytes, do what you want" leads to a fragmented world of low quality idioms. And hence gets judged poorly.
The problem with not doing this is that the operating system is imposing some arbitrary standard on data that may not have even come from the same system.
If the data you get is JSON, there are utilities for parsing JSON. If it's XML, there are utilities for parsing XML. It could be a PDF and you want to convert each page to a TIFF and then tar them into an archive. The file came over the internet; it doesn't care about your system's encoding defaults.
Is that why 80% (at a guess) of shell scripts don’t handle spaces in file names correctly?
If I send your system something obscure like an SDIF audio file you're not going to have the tools to do anything with it unless you can find and install an SDIF library.
What if the OS could do this automatically because a typed schema and standard API was built into the SDIF (or any) standard and installable in the same way code dependencies are?
Which is the point. So then you find and install an SDIF library. The OS can't reasonably predict and support every possible file format that anyone will ever come up with in the past or future, so the answer is to be flexible and use different utilities for different formats.
> What if the OS could do this automatically because a typed schema and standard API was built into the SDIF (or any) standard and installable in the same way code dependencies are?
The problem with this is that what you can do with a file depends on the format. You can split a PDF into pages; you can't split an SDIF into pages. An SDIF might have channels; a PDF doesn't have channels. You could call each frame in a video a "page" and try to compare them with a PDF, but the video "page" is going to be a pure individual image whereas a PDF page might have text or multiple individual embedded images.
Everything is so specific to the format that you're going to want format-specific utilities to operate on it that take into account the characteristics of the format.
At best you'll sometimes have multiple formats that are largely "the same" interface-wise like PNG and JPEG and then you would want to use the same interfaces to access both, but that's what already happens. You use e.g. ImageMagick when dealing with both PNGs and JPEGs because it supports both. But it doesn't support SDIF because you need a completely different interface when dealing with audio.
And file formats already have magic strings in them that identify what kind of file they are, which is about the extent of the information the OS could provide you about the file without assuming anything about what's inside it. The schema is already implied by the header magic.
The whole point of OP is that it should be easy - or at least possible - to access contextual meta-information. Because this makes it easier to to build tools that work with meaning automatically instead of relying on unreliable, partial, and/or broken manual context handling based on unstated or implied assumptions.
Now, of course it would it be better if each program formally described the schema of its output and the system provided tooling to act on this information, but being able to string together uncooperative programs via their common denominator (i.e stream of bytes), is powerful and the unix way.
I can't disagree with this more, Unix is broken in so many ways and its designs continuously cause me pain. I would not describe a modern Linux or BSD as "simple", maybe the core of it is, but to actually build a modern OS there are so many layers on top of it. And when you do that, the crumminess of the foundation really starts to stand out.
Operating systems shouldn't get too hung up on supporting applications or browsers, instead they should focus more on serving humans.
Common operations that humans need should be standardized across the operating system.
(bottom left corner)
I do NOT want to go back to the times when we had to write and ship individual drivers for each graphics card, sound card and peripheral we wanted to support in our software.
Could you imagine having to select the physical port, IRQ and DMA channel of your sound device for each app that wants to output sound? Because that's what had to be done in DOS and screens like this https://flaterco.com/kb/audio/PCI/DS-XG_IRQ_error.png
were commonplace. Good enough if you had well-defined devices like SoundBlaster(Pro) or GUS that you could auto-detect without crashing the PC, but a living nightmare with clones that didn't quite behave the same way.
So unless you only plan to support a single set of well-defined hardware, device drivers are a must.
This 1000x - I remember this time and it was quite painful
>Could you imagine having to select the physical port, IRQ and DMA channel of your sound device for each app that wants to output sound? Because that's what had to be done in DOS and screens like this https://flaterco.com/kb/audio/PCI/DS-XG_IRQ_error.png were commonplace. Good enough if you had well-defined devices like SoundBlaster(Pro) or GUS that you could auto-detect without crashing the PC, but a living nightmare with clones that didn't quite behave the same way.
I also remember the application working one second, unrelated hardware change (1), and back to the drawing board all over again
We are currently in this world. NVMe, USB, etc. NVMe for example is "just" DMA'd ringbuffers of requests and responses.
One challenge is that vendors tend to want to have "special features" that go beyond the standard command set. SCSI and SMART are examples of that pain.
> The protocol should support the ability to upload custom “shaders,” whenever this makes sense, to tweak or extend the default behavior.
This part is crazy complex and not very likely to happen. "Shaders" in the context of GPUs can work because it's a very large & expensive component and very vendor-specific already. That approach won't work for a chip that costs $2, with the additional burden of needing to have the same programming interface for competing vendors.