Cthulix: a Single System Image construction kit that is currently in development | Hacker News Reader

Cthulix: a Single System Image construction kit that is currently in development

cthulix.com

107 pointsteleforce30 comments

>Cthulix is all about bilateral contracts, and hence discourages Files and multiparty filesystems.

Sanity at last. Default access to a filesystem is the single biggest sin of Unix. It was a reasonable thing to do in the 1970s, but not since.

>Rather than having a filesystem, you would create an ETL that had multiple inputs (one from each producer) and multiple outputs (one to each consumer).

I strongly suggest a capability based approach to have read/writeable resources... but maybe that's not really the best way...

>Future work: build a gridapp that makes it easy to create bilateral File-like things.

Ok, I'm willing to ditch random access files because this is a multithreaded parallel world. A gridapp that allows for versioning of the state of a "file" and appending mutations that are enforced in a consistent manner, or are guaranteed to become consistent at some point, would be a very good thing. Think of the really old "Paradox" database that was internally append only and versioning, but appeared to be read/write when necessary.

---

I'd be happy to help write an implementation that spoke to Cthulix in Pascal on Windows/Linux/DOS/WebApp to have a second source in a completely different source tree.

Kah thu licks - right?

> Sanity at last. Default access to a filesystem is the single biggest sin of Unix. It was a reasonable thing to do in the 1970s, but not since.

Novell Netware had what seemed to be a reasonable compromise some 30 years ago, called "Trustees" - it had a permission system (ACLs, if I remember correctly), but it was magical in the sense that you only needed to manage the ACLs of the leaf nodes; If you had access to a file in a path 7-directories deep, then you would automagically see the entire path (but only files you had access to in each directory); and if you didn't, neither the leaf not the 7 directories leading to it existed for you.

A billion times easier to manage/reason about than the NT or Unix mess.

> Default access to a filesystem is the single biggest sin of Unix

Doesn't the widespread adoption of containers (Docker/OCI/K8s/etc) solve most of this 'sin'? You can make a container's root filesystem read-only (not always done, but recommended for maximum security), so its contents is always the immutable image from which the container was started. You can also provide the container with additional read-write volumes if it requires them (such as to persist state). In systems I've worked on, most of the containers are stateless, and they store their data in Cassandra/Postgres/S3/whatever, and it is only containers which host Cassandra/Postgres/etc themselves which have any persistent read-write volumes attached.

Even without containers, managing permissions across some apps is not that complex for most workflows. Now try to introduce human users and all hell breaks loose unless you have actual ACLs.

This part of a sibling comment deeply resonated with me:

> If you had access to a file in a path 7-directories deep, then you would automagically see the entire path (but only files you had access to in each directory); and if you didn't, neither the leaf not the 7 directories leading to it existed for you.

> Now try to introduce human users and all hell breaks loose unless you have actual ACLs.

Most apps I've worked with don't use filesystem ACLs for user-facing security.

Suppose you are building a document management system. You might store each document as a file on a filesystem–but you probably won't store "who can access what document" as filesystem ACLs. You'll probably just have a filesystem ACL giving the application access to all the documents, and then enforce user-level access in the application code, with the user-level permissions stored in the application's database.

Human-managed permissions turn into a mess no matter what you do. But enforcing them at the application level, you can make the implementation actually understand some of the business rules which drive them. In a B2B setting, you might have a business rule that for each customer account there is a list of authorised employees allowed to work on that account, plus some workflow process to get approval to permanently or temporarily add an employee to that list. A lot less mess if you build a database and UI around that business rule, than if you just try to store that directly in filesystem ACLs.

A document management system can even export the documents as a network filesystem (WebDav, CIFS/SMB, NFS, etc) – but it doesn't have to properly implement filesystem ACLs in doing so, if users want to manage permissions you force them to use the Web UI instead.

> Most apps I've worked with don't use filesystem ACLs for user-facing security.

Same here! Although i believe that's a shame we have to pile layers of "security" (or holes) on top of one another. I long for better tooling for FS ACLs and journaling!

I see nothing sane in an OS without an FS.

It's an OS without a default permission to access a file system. Somewhere there must be a place where state is persisted. More than likely there will be a file system.

You could certainly write a Unix/Linux/Windows program to run given a set of file handles instead of permission to the whole file system.

Pipe based utilities don't really need a file system, just the pipes.

The author seems to have worked exclusively on trading systems, which probably explains why the section that claims “This is how most businesses build software” bears little resemblance to how I’ve seen software built.

From the site:

Q: "What is released so far?" A: So far, there has been no release of code.

We've changed the title to the language used at http://cthulix.com/index.html, which makes this clearer.

That is an improvement, although on the status page, they say,

"The principal developer signed with a commercial venture in early 2021. There will be a period of inactivity for this project."

So, I'm not sure if it is in development...

I'm reading this and I don't understand how what it is proposing is different from what people are already doing?

> 2. Deployment. They pass the software to another team, who are responsible for deploying it.

Use ArgoCD to deploy to K8s, and then make the developers who write the code responsible for production deployment. To deploy to production, they just get their PR approved by the right managers, and then at the appointed time click the "merge" button and ArgoCD does the rest. And part of getting approval to deploy to production, they first have to demonstrate the PR can be successfully deployed (again by ArgoCD) to a pre-production environment (pre-prod environments are isolated from each other by having a separate K8s namespace and ArgoCD application for each; and the pre-prod environments are isolated from production by putting them in a separate K8s cluster.)

I've seen firsthand that model work. What is Cthulix adding to it?

There is a section "What about Kubernetes?" – but it doesn't clearly explain how Cthulix and K8s differ, and what problems the author sees with the K8s model, and how Cthulix will avoid them. "If you use K8s like that, it will lead to the algorithms-first mess warned about early in this paper" is a very vague statement, and even supposing it were (in some sense) true, it isn't clear why the solution to that "mess" has to be a whole new system as opposed to just a different way of using K8s, or some evolutionary changes to K8s (new operators or whatever)

We all like GPUs. Our author thinks they’re sufficient reason to stick to Linux. They’re not liable to go away, and continue to grow in power and importance. And a shiny new A100 can slurp a gigabyte of data a second, I think it was? You can buy eight of them in a single server.

But the rule of thumb they tell me is: If you try to use CPUs to move data to all these GPUs, your performance is going to tank. If at all possible, you want tech like RDMA, something that throws bits directly from your filesystems to your GPUs. This will only become more important as GPUs get more capable and data hungry.

Can Cthulix actually control enough I/O to build the future of distributed computing? The approach described wants to explicitly minimize filesystem use in favor of message passing. I’m not sure how that’s supposed to be implemented exactly? but there are only a few ways to get the I/O performance you want for the big heavy GPU workloads, and this doesn’t naively sound like it’s one of them, but maybe that’s because it’s a big question mark.

Zero-copy message-passing is a thing for a long time. DMA from a NIC to a RAM buffer, from the same buffer to "disk", or to GPU, etc.

I'd rather have a capability-based distributed system that is agnostic about persistence as long as capabilities are, well, capabilities.

A friend of mine is working on Syndicate[0], and I find it interesting. Not really an OS metaphor; more like distributed, polyglot actors with shared memory secured via capabilities. I think of it sort of like the OSI model's missing session layer.

Also, if I were proposing a distributed system to my boss, I'd probably prefer a reference to "Syndicate" over "Cthulu."

[0]https://syndicate-lang.org/

This is very, very well written. It’s incredibly hard to frame something like this so well in crisp, engaging text that reflects against common ideas and differentiates at the same time.

This felt eerily like something I would design. But the more I read, the more I realized this won't work for most people. There are a few things that sound nice and a lot more things that only work for specific people in specific scenarios. There are a whole lot of "wait... what?!" moments in here, but then I realize it's just a fancier MPI cluster and it makes sense.

This looks to take on the responsibility of MPI, plus a bunch of extra stuff, and as far as I can tell it is run by just one person (with no public code releases). As such,

> Pitch

> (Cthulix <=> MPI) is as (Unix <=> Multics)

comes off as quite ambitious.

So, single system image implemented mostly in a userspace runtime? Which requires writing your applications for it, but is probably easier and more reliable to implement.

Chutzpah -- I like it!

How is it pronounced?

I assume it’s pronounced like Cthulhu, but with an ‘ix’ sound at the end instead of ‘u’.

> I assume it’s pronounced like Cthulhu, but with an ‘ix’ sound at the end instead of ‘u’.

I imagine most people don't know how to pronounce Cthulhu... I only learned how to pronounce it about 30s ago when I Googled it. Apparently it is not "Cha Hu L'Hu"

Please refer to the Al-Hazred book that explores the question in detail.

Exercise caution.

It's a little hard to find, however. In the US, you can probably find a copy at Harvard and of course Miskatonic University.

> Q: "What is released so far?" > So far, there has been no release of code.

bummer, I was hoping for some examples

What the hell is a “quant”? :P

They are Bach, we are the orchestra.

…except instead of scoring symphonies, quants algorithmically encode rules for trading Jimmy Choo future default swaps which they think will eke out more picodollars per transaction than the other funds, which are then dutifully executed by the infrastructure teams.

It’s a finance term, short for quantitative analyst

https://careerkarma.com/blog/how-to-become-a-quantitative-an...

It is a fintech term for a person (these days an entity) that can use a trillion bazillion Loonies in a micro second

Reply on news.ycombinator.com