It's Time for Operating Systems to Rediscover Hardware

Plus, if we ever are to transition to the world of microkernels (something that OS researchers have been pushing for decades now), we're gonna need a massive amount of people writing drivers just to get us to the point where things are barely functional.

That class bordered on legendary. Awesome that you got to TA it. I still have an irrational fondness for AFS.

https://news.ycombinator.com/item?id=17692499

That said, BIOSes can also emulate PS/2 with USB (no doubt via SMM). This is usually a setting called "legacy keyboard/mouse support".

[1] The notable exception being Apple. Think Different. https://news.ycombinator.com/item?id=12924051

"Hardware" (read: device drivers) is not notably complicated IMO.

That's not to say they're simple, but driving devices from a software point of view is pretty similar to interacting with other software. You read the spec (APIs), write code to set up data structures or registers a certain way, parse responses, etc.

Writing a modern full fledged USB stack is very complex, but would not be more complex than writing a modern full TCP/IP stack for example.

A lot of device programming models have gotten simpler too. Device drivers used to be notorious deep voodoo magic e.g., in cases of the IDE disk driver, but that was not really "complexity" of the software logic so much as hundreds of special cases and deviations from standards or odd behavior in all manner of IDE controllers and devices. To the point where the real wizards were the ones who had access to internal data sheets, errata, or reverse engineered firmware from these devices, or otherwise spent countless hours poking at things and reconstructing quirks for themselves, probably bricking a lot of silicon and spinning rust in the process.

But systems and devices are now getting to the point where things don't work that way anymore, silicon power is so cheap and firmware is on everything. The NVMe device for example sets up pretty simple packet-type command queues and sends requests and receives responses for operations - query device information, perform a read or write, etc). It's not quite that simple, there are some quirks and details, but it's quite a lot like writing a client for a server (HTTP, perhaps). I think other device interfaces will evolve toward cleaner simpler models like this too.

One exception to this is GPUs of course. The other thing is with moore's law continuing to slow down there will be increased incentive to move more processing out to devices. It's long been happening with high end network devices, but with technologies like CXL.cache coming soon, I would expect that trend to keep ramping up and come to other accelerators (crypto, AI, even disk).

So.. it's a mix. Things are definitely getting more complex, but AFAIKS the complexity of interacting with hardware is not increasing at a much greater rate than the complexity of interacting with other software. And, as always this complexity is made possible by abstractions and layers and interfaces. It's not clearly exceeding our ability to cope with it.

Yes, it’s hard to do research if you start at “let’s design a kernel from scratch,” but you’d never need to do that. You can just hack Linux or a bsd, or even use something like bpf to extend it.

The thing that annoyed me about 410 is that when I switched to Linux I realized that pusha/popa didn’t matter at all. It was a good course for writing reentrant C and learning the very basics of hardware, but the really hard stuff is in the weird dynamics of memory management on NUMA systems and work conserving io, which you can’t get anywhere near if you are starting from scratch.

For the complete USB stack, that's very likely. However, for minimal code needed to read keys from keyboard USB shouldn't be that bad.

These students only need USB 1.1. They don't need to support bulk transfers (USB protocol has 2 distinct transports, one low latency another one high throughput). They don't need to support USB hubs between computer and the keyboard. The only device type they need is HID.

USB consortium wouldn't allow to call that protocol USB due to the missing features, but it's good enough for education purposes.

The whole combined experience just makes things run so smoothly and allows security issues to be patched since Apple has the resources that maintain each part internally.

One method could be, say, extending microkernel capability systems to incorporate these remote processors. Or perhaps even the ability to provide WASM or ePBF control blocks to customize the power management system. Though its true that'd require more engineering resources.

Actually USB is somewhat like this in that there's a hardware api and spec that the OS knows about. Linux has troubles with even that in my experience (often requiring a reboot to fix.

You have good questions in your comment, and I think he is calling out for searching answers to them and other similar questions. Apparently, the operating system researchers haven't been doing this enough, or at all. His speech is a lament for this state of affairs, and a call to get excited about finding out what benefits would be gained by doing things (very) differently.

But that's just my take, and I'm not working on operating systems research.

On the radio side of things, Dash7 firmware will run on many Lora devices, meaning you don't have to use the proprietary Lora software.

I think a “true” OS would run its code on all these processors, implementing both sides of the communication protocols connecting these pieces of hardware into a computer.

For one, such hypothetical OS may upgrade or replace these communication protocols with OS updates, or because user changed some system preferences.

This also allows to re-configure these things based on runtime information. For instance, when AC power is disconnected from a battery-powered device, it makes sense to do stuff slightly differently, like stop all processes on all CPUs who are polling to get events faster, and switch to slower but energy efficient interrupts.

Actually, come to think of it extending DTS syntax to describe the pointer swizzling the OP video mentions. Then both OO'es could derive correct physical address handling.

1: https://www.linuxfoundation.org/blog/the-power-of-zephyr-rto... 2: https://docs.zephyrproject.org/latest/guides/dts/index.html

The problem is the proprietary nature of this stuff and the fact that a lot of it is outside of the scope of scrutiny that goes into Linux. It's held together with what basically amounts to glorified duct tape. It's complicated. It has weird failure modes and occasionally this stuff has expensive issues. Like for example security issues.

Very relevant if you want to build an open source phone platform or laptop. Not impossible; but it requires addressing a few things. Like connecting to a 5G network without relying on Qualcomm's proprietary software and hardware.

The real solution is not merely integrating these things as is and trying to reverse engineer them but engineer them from the ground up to work together and make sense together. That larger, open operating system does not really exist. And many proprietary equivalents on the market right now have an emergent/accidental design rather than something that was designed from the ground up.

I remember how it was with programmable GPUs.

When programmable shaders were introduced in GeForce 3 (2001), nobody really understood what to do with them. The consensus was that it’s incredibly hard to write custom shaders. OpenGL consortium refused to support shaders for almost a decade (GLSL 1.0 released in 2008), that’s one of the reasons of the success of Direct3D https://softwareengineering.stackexchange.com/a/88055

Yet look at them after 20 years of evolution. People no longer coding shaders in assembly, we have quite a few higher-level languages for them. People have built better APIs to interface with programmable GPUs, both low level close to metal (D3D12, Metal, Vulkan) and high-level (CUDA, TensorFlow, game engines). People built debuggers and profilers for these shaders. Overall, people embraced programmable GPUs, leaned to use them, and the results are amazing. Visual quality of games and other real-time 3D skyrocketed, despite the increase of pixel count and density in the displays.

I can see how the same path is possible for the rest of the programmable chips and pieces inside modern computers.

> and the benefit is very slim

Better products in very broad sense. For instance, when idle, a smartphone OS could power off not just the screen, but also CPU, RAM, and a few other components, and downscale itself to the slowest and the most power efficient core in the system.

I don’t know though if I fully agree with the characterization that Linux has been shuttled off to the corner as chip vendors work around it. It’s more that the computer has actually always been a distributed system and OS/hardware developers have ignored that for a very long time. You wouldn’t say that Linux isn’t the OS for distributed clusters even though a lot of work goes into making those clusters run and that all typically runs in user space outside the OS boundary.

It’s possible that there’s a better model out there that manages to unify things but I’m skeptical. All this code runs on different chips with different clock speeds. Code also resides in drastically different memory spaces and that’s unavoidable - an M3 might be able to load the firmware over PCIE, but it’s not executing out of main memory. Additionally these chips typically have drastically different cache coherency properties.

Now maybe we need to revisit this all holistically, which I think is the actual pitch being made and one I can support. That needs to look like defining interfaces for how this stuff works (& yes, making it possible to integrate into Linux) and getting chip vendors to adopt. Sunk costs are real and trying to rework not just HW architecture but also the entire SW ecosystem can be a dead end. I’d certainly be keenly interested to hear the speaker’s ideas. He’s certainly far more knowledgeable about this space than I am.

If Linux is running inside a sandbox, it can't secure anything.

Or that you open two programs at once and they try to access the same device at once causing some kind of error and likely data loss or system instability.

I don't even want apps to directly have access to the file level interface of my storage but rather an extra abstraction which lets me pick which files it can access.