Hacker News Reader

> I want to try replacing all of that with a simple, unified API which lets you say "tell me the state of <kernel resource X> and tell me when it changes".

Fuchsia's "Signals" are designed in that direction. You get a "handle" to a resource, and the resource has a 32-bit integer with signal bits that you could subscribe to. The meaning of each bit is different for each type of resource.

(In general operating system terminology, "Signal" means just a message without any payload that would have required any memory allocation. There are many ways to do it, not just the Unix way. Message ports and other abstractions are sometimes built on top of types of signals)

Oh I didn't know that! Thanks - I'll take a look.

> In general operating system terminology, "Signal" means just a message without any payload that would have required any memory allocation.

Do you have a link for this? It's the first time I'm seeing signals described in terms of lack of memory allocation. I always thought the word "signal" basically meant user-mode interrupts - asynchronously hijacking a thread.

> Have either database-like transactions or (at a minimum) write barriers for file operations.

Go one step further and have the kernel expose a much more structured view of storage to user space by default. 99.9% of user space programs I've written don't want to deal with file IO, they want a database, or some other way for arbitrary in memory objects to be persisted and shared between processes. There's so much room to improve on the 1970s era IO primitives we use by default.

Ironically my understanding is that several operating systems around that time did do that, then Unix came and said "a file is just a bag of bytes in a hierarchical namespace" and we've seemingly never reconsidered

I can imagine that there's a major lack of flexibility inherent in a record-based filesystem. Nowadays we have file formats that introduce a record format on top of the "bag of bytes" for when you want records, but you aren't required to have records of any particular shape and size (or have any records at all) if they don't work for your needs.

Notable major player here was IBM on the AS/400, you had libraries containing objects of various types (such as PGM/Program, FILE/'file', *DTAQ/Data Queue, etc).

'file' objects were effectively database tables, and even things like source code for programs were stored as such.

Eventually IBM added the IFS (integrated file system) to provide a more traditional POSIX-adjacent filesystem that was in fact just a bunch of bits, and it feels weird and out of place with the rest of the system as a result.

One problem with 2. is that subscribing to changes isn't always good enough, since if you go to act on the changes things may update out from under you regardless. Like, if you react to filesystem events, if a new file is created you might have it get deleted before you even register that it was added and do whatever you wanted with it.

I feel like it should be possible to say "I want to reserve access to this resource when something important happens on it". Or just "I want to reserve access to this resource". Hopefully eliminates TOCTOU races. I guess then there have to be different modes to balance scalability/flexibility.

You can definitely do that, it just means that userspace can now stall access to things unless you're very careful :) Like, one model (often used by EDR, which is why everyone hates them) is that every operation must be "authorized" so it doesn't go through until some evaluation happens. But this means the kernel has to ask a program for every operation, which is slow.

What's EDR? But yeah, I recognize that there need to be limits. I haven't really thought it through yet, heh.

EDR stands for Endpoint Detection and Response or Remediation.

Think modern antivirus. Less periodic scans and more continuous monitoring for anomalies.

When people complain about antivirus and endpoint security applications slowing their computer down, this is what they're talking about. This leads to user-visible delays (which, to them, seem to be random) as the reserver does what it wants right after the user has done something to the file. Many (most? ALL?) apparent Explorer stalls are really Windows Defender locking and scanning the file right after you did something to it.

Yeah good point. I don't want write stalls because of subscription requests. For things like filesystem watching (for compilers), it'd be enough to have a form of snapshot checkpointing. (There were these FS events within 10ms [..]) / (Then the file contents was XXX) / ... (Then there were these FS events) / and so on.

Its possible you'd end up in situations where the OS would need to retain old file contents if there's both a pending read from a subscriber and subsequent writes. But in most cases, this is fine.

Going back to my original comment, this is the sort of reason I'd like to try this in my own OS kernel rather than try and solve all these problems immediately in linux. It'll give me a chance to try out a few different solutions and see if I can find something that works well.

It should be possibly to speculatively execute code that's accessing resources and only allowing transactions on system resources to commit successfully if other transactions haven't invalidated them.

Meaning a process could start loading and reading while queuing writes and only writes would block while another process is locking access to the executable to scan it. If the scan succeeds the writes are committed and if it fails it could revoke execution privileges and all the work is undone.

Lot of hand waving there, but when you're not committed to existing file system semantics there is a lot of room for innovation.

> It would also be interesting to try making all userspace programs be wasm bundles and run them in ring0. But I suspect that will run slower on modern hardware compared to using native binaries and context switching.

If you have all of your programs reside in the same virtual memory space (which is the implication with running them in ring 0), then you get a unique performance advantage in that you don't have to invalidate the TLB. I have no idea how much the amortized performance impact of virtual memory is, unfortunately, so the gains might be trivial, but who knows.

Yeah - if you run things in the kernel you don't need to do TLB flushes or context switches - which can be very expensive. But thats all offset by the fact that wasm code runs slower inherently, because it needs to bounds checks everywhere.

I suspect that programs which make a lot of kernel calls would run faster in the kernel in wasm, and programs that are more compute / memory bound will run faster as native processes. It'd be fun to play around and find where the tradeoffs lie for different applications.

Sorry for being daft, but doesn't this imply that TLB is now shared between all processes? Which means that the cost of flushing the TLB is amortized across context switches? Processes are gonna evict TLB entries from other processes at unpredictable intervals.

I'm not sure if I understood you correctly.

If all programs run in ring 0, you can just run them all in the same virtual memory space. Then you don’t need TLB flushes at all. Of course, one bad program could bring the whole system down but that’s the price you pay.

But if programs were written in wasm, that problem goes away. Wasm prevents bad memory accesses by doing memory protection in software. (Every array or pointer access is bounds checked).

> I'm increasingly tempted to try my hand at making a toy OS - or at least a kernel.

I strongly encourage you to do this!

I think writing your own OS is one of the things every dev should do at least once, just for the educational value. The others are writing a game and writing a compiler and linker.

Some of the issues can be improved "hackily" by for instance giving a database software exclusive access to two discrete disks - one for the db file, one for the journal.

Why is this "hacky"? I've long wondered why databases need to be on filesystems, other than simply convenience. If you think about it, a filesystem is, itself, basically a database (not a relational one of course). So it seems like you could improve performance a lot by eliminating the filesystem layer and going straight to direct disk access.

Also, how this would be done would probably change depending on whether you're using a spinning-rust HD or an SSD, or a RAID array.

Lots - maybe most - applications need to essentially run their own mini database to store user data. I obviously don't want a partition per application, or per word document.

Abstracting away your hard disk is literally the job of your filesystem.

I don't mean that some little SQLite DB for your web browser's storage and settings needs its own partition; that's obviously going too far. I mean for really huge databases, where an entire server is dedicated to running that DB and it has high performance needs.

Fair enough. But I am including all of those little databases.

I agree with you - you could hack around this for a big database by putting it on a dedicated partition (even though you shouldn't need to). But I'm also thinking about applications with little databases that need to live inside the filesystem. They shouldn't need to embed sqlite in order to survive a crash.

A different syscall API would allow you to write a small, fast, efficient database engines (like redis or mysql) without needing all of the complex, performance gobbling tricks that these databases currently have.

A better filesystem API would help databases everywhere.

last time I looked at this, and it was a long time ago, it was around 10% and workload dependent.

if you make direct and locking more robust, and allowed an application to turn off the kernel cache, you would only really need to interact with the filesystem after open in order to change your extents.

+1. this is clearly a place where we can have nice things and performance

For server applications, the trends of virtualization and microservices mean that having "machine" dedicated to running a single application is a very common scenario, and also for server VMs running a bunch of apps it is not obvious that having a partition per application is an issue, it would be trivial for the VM initialization scripts to automagically configure a dedicated partition for each DB.

Having a OS+filesystem running a virtual OS+filesystem running a database has all kinds of redundant caching, and we could systematically skip a layer here.

>I've long wondered why databases need to be on filesystems, other than simply convenience. [...]So it seems like you could improve performance a lot by eliminating the filesystem layer and going straight to direct disk access.

Both Oracle RDBMS and older versions of Microsoft SQL Server had the option to use "raw disk devices" / "raw partitions" instead of the filesystem. It had some obvious justifications such as avoiding the "wasteful double buffering" of the file system cache being redundant to the database's cache and avoiding "unnecessary" extra i/o abstraction layers.

Microsoft later got rid of that option because it wasn't worth the tradeoff of a small performance gain while losing the easier management aspects of NTFS file system.

Oracle also had (has?) the capability to split up database volumes on raw disks based on I/O load.

SQLite essentially achieves this as well by keeping the entire database in one file. It’s even faster than the filesystem in some cases: https://www.sqlite.org/fasterthanfs.html

I'd like to go the other way, and have a filesystem that leverages a proper database for its metadata.

I can search millions of records in a split second in SQL. But searching records on my hard disk takes enormously more time. The solutions out there for speeding it up (like Windows Indexing service) rely on asynchronous indexing that's patched on top of the filesystem instead of integrated realtime within it, and are subsequently brittle (in my experience), slow down your computer at inconvenient times, and too often aren't instantly up date.

Being able to have stored procedures execute when changes trigger would be another interesting aspect that could take the place of event filters and file watchers (especially if critical portions could be configured to happen sequentially & atomically, akin to IRQ handlers).

I feel like databases have gotten lots of love over the decades and are super optimized (and by this point are pretty dang reliable). Filesystems feel like they lost their time in the spotlight and have floundered with less innovation.

Hardware that calculates CRC's of blocks in realtime would also be awesome (imagine what it would do for sync). The ironic thing is your HDD already does this, there's just no way to tap into that information at the software / filesystem driver level.

>I'd like to go the other way, and have a filesystem that leverages a proper database for its metadata.

Microsoft made an attempt back in 2003 at creating a "smarter" file system with rich metadata based on a real database engine (MS SQL Server) ... but eventually abandoned the project: https://en.wikipedia.org/wiki/WinFS

WinFS wasn't necessarily going to completely replace NTFS but Microsoft did have ambitious plans for its integration into everything.

It's interesting that other platforms like Unix/Linux/Apple that were designing "next generation" file systems like ZFS/ext4/APFS after the failed WinFS project had a chance to try adding their own WinFS smart metadata features to but they didn't.

The later fragmentation of user data being spread around in the cloud such as AWS objects, Apple iCloud, Backblaze backups, etc instead of every file saved on home computers makes a "universal rich metadata index stored in a database engine" to be a cross-platform standard by all operating systems less realistic today than back in 2003. Therefore, the core os file systems remain a "dumb blob of bytes" with a separate layer of metadata index db in vendor-specific schemes like Apple Spotlight db, Windows Search in ".edb" file, etc.

Yeah they totally missed the mark with WinFS, in my opinion. It was too exotic/foreign with no clear adoption path.

All I want is dumb files with something like SQLite replacing or augmenting the allocation table. Then sprinkle on some new capabilities incrementally.

When WinFS was canceled, I started my own pet project to build a new kind of file system that totally changed the way file metadata was stored and managed. I built an object based system that could easily handle hundreds of millions of files within a single container (what I called a pod) and do searches much like how you would search a large database table.

The 'file table' contains a 64 byte record for each object and has some very unique characteristics that make searches incredibly fast (e.g. it can find all the photos in under a second, even if there are 20 million of them among 200 million files). You can also attach various metadata 'tags' to each object and find matching objects based on them.

The project is in open beta and anyone can download it and try it out on their own computer.

https://www.didgets.com

The reason indexing services don't work synchronously is because it would cause file IO to get far slower than apps expect, and that can break user interfaces and cause unexpected slow downs. There's also no real reason for it to be synchronous in most cases.

You'd probably enjoy reading about the Be File System: https://en.wikipedia.org/wiki/Be_File_System

It has relational database-like semantics for metadata.

Do it. Ignore whoever says it's too complicated.

You probably won't supplant Linux with your toy OS for your own research, but for a software engineer, writing a OS is the closest thing we have to the exhilarating feeling of seeing your code affect real hardware. You haven't lived until a bug of your own causes a triple-fault exception that causes a literal system shutdown.

I started this career writing my own tiny OS in my teens (https://github.com/1player/klesh) and for 20 years I've been wanting to go back, with all the experience I have accumulated.

Just don't make it another UNIX, please. We have enough of those already ;)

> Just don't make it another UNIX, please. We have enough of those already ;)

Hah thanks for the encouragement! Don't worry, there's no danger on that front. I'm pretty thoroughly bored of the "everything is a file" motif.

I do like unix's idea of making large programs out of small programs that can be composed together. But unix's raw byte streams make a terrible messaging bus.

You're on the right path! My area of interest is actually making messaging (a-la-Smalltalk/Erlang) the root idea everything is built upon, rather than the file stream. The file metaphor made sense in the 1960, not in an ultra-networked world of heterogeneous CPUs. IMO, communication should be the basis of computing.

Which also means inventing a language to build this novel paradigm. As Dan Ingalls said, an operating system is everything that doesn't fit in a language.

I am looking forward to finding the time for this long and quixotic quest :-)

You might enjoy http://synit.org/, perhaps. Heavily inspired by both Smalltalk and Erlang. (It's my own wee quixotic project btw)

That looks very interesting, thanks for sharing! Somehow I had never heard of it. Time for a Show HN?

Thanks! And good idea. The ideal time would have been early 2023, because things have drifted into bitrot a little already. I'll tidy up and see about showing it off hopefully in the next few days.

Why a new language? Why not build an OS around Erlang?

I present Crazierl: https://crazierl.org/

My crazy hobby OS vision of boot to Erlang. I've been working on this for a couple years now, recently made it public as part of a PR to v86 [1] (which is amazing, by the way). 32-bit x86, has drivers for virtio and rtl-8168 (but not ne2k, so no networking in v86, sadly), works with SMP. Probably not better than running beam as init with a FreeBSD kernel, but was more fun?

[1] https://copy.sh/v86/

> You haven't lived until a bug of your own causes a triple-fault exception that causes a literal system shutdown.

On the OS I work on, I deliberately added a triple fault trigger as a fallback method of rebooting the system if the "normal" methods fail, which I have seen before.

To anyone looking to play with writing their own OS, definitely just do it. It's a lot of fun, and there's almost no limit to how much you can learn. With QEMU and bochs, and VirtualBox, etc... it's not that hard getting started. If you want to play around without starting from scratch, but find Linux source too complicated, give OpenBSD a look. Not that Linux is that hard, but it takes some getting used to how they do things. I find OpenBSD considerably more approachable.

>but for a software engineer, writing a OS is the closest thing we have to the exhilarating feeling of seeing your code affect real hardware.

Unless you want to muck about with toy robotics or home automation.

> The current filesystem semantics are terrible for databases

There was at least one Unix startup (IIRC it was Tolerant Systems) in mid 80s that applied transactional semantics to filesystem operations. I did some contract work for them and found the performance was absolutely abysmal and the system had some deadlocks. Now may be their implementation was not great but I suspect the converse is also true: database semantics are terrible for filesystems :-)

There was at least one database from the mid 80s with abysmal performance and deadlocks too. But plenty of modern databases are fast and correct. The fact this one startup 40 years ago didn’t solve the problem doesn’t say a lot about whether I’ll be able to. Honestly the fact there are some dead bodies on the mountain makes it more fun to give it a go and see how I do.

To be frank, judging by the code I’ve seen over the years, most engineers don’t have a lot of experience or skill at making software run fast.

Proof is in the pudding etc.! I'm certainly curious how well your design & implementation will turn out. Good luck with your project!!

I am very interested in this tutorial for building an OS for the Raspberry Pi in Rust: https://github.com/rust-embedded/rust-raspberrypi-OS-tutoria...

I'd love to try it out when (if ever) I have the time.

a great place to start is https://wiki.osdev.org/Expanded_Main_Page

So soft updates and kqueue? :)

I'm not sure you mean by soft updates but - kqueue is really just a messaging queue. I want a data subscription system, so I can get a kernel object and get a stream of updates which can be applied to the snapshot in sequence.

kqueue is very limited in terms of what you can subscribe to, and things like filesystem watching usually requires subsequent filesystem calls to see whats actually changed. And that in turn introduces race conditions if the data is changed again before your fs syscalls make it through.

I think there are cleaner APIs out there. I'd like to see if I can find one.

This seems a total waste of time when you could be learning LLMs

I think macOS/Darwin does have write barriers like you want.