Hacker News Reader

Comment by catern | Hacker News Reader

Distributed software, which accesses and depends on remote resources, is harder to run than local-only software. I'm working on making it easier to run distributed software for both programmers and end-users.

See the main article describing my current vision: http://catern.com/integration.html

My most significant project in this area: https://github.com/catern/rsyscall

My website in general: http://catern.com/ (happy to collaborate on anything on there)

And my current plans: http://catern.com/plan.html

If you're interested, feel free to contact me using the information on my website.

View on HN

I read this comment 3 times and skimmed both links but I still have no concrete idea what you're building.

Can you say what left you confused?

I'm surprised, I thought https://github.com/catern/rsyscall is pretty self-contained and self-explanatory (if unclear on what exactly rsyscall is useful for), and I thought http://catern.com/integration.html is quite detailed (perhaps excessively so).

I understand the problem and the use-case, but what does your library provide? Running processes remotely? Supervising remote processes? Transparently running Python functions in remote processes?

>Running processes remotely? Supervising remote processes?

These things. This is mentioned in http://catern.com/integration.html#thread

So this offers capabilities like `multiprocessing` but possibly across the network, e.g. queues and shared memory? Can it spawn the remote processes too? With SSH, or its own system?

You keep pointing me to that document that you feel is compelling, but all it does is explicitly run a Python function in a local thread... I know it's meant as a shortcut in the example but it's hard to infer what the real capabilities are, when what you're doing is not much more than what `trio` or stdlib can do.

>Can it spawn the remote processes too? With SSH, or its own system?

Yes, with rsyscall, is that not clear? It's pretty explicit in http://catern.com/integration.html#thread

>but all it does is explicitly run a Python function in a local thread...

That's not all that's happening. I think you might be pattern-matching what's happening in the article to something you've already seen, when in fact it's something novel.

It's clear that it "may operate on a local or remote host" and in this case you run a local thread. Just how much is actually implemented is not clear at all. A lot of frameworks are written to be "extensible" but the extensions are left as an exercise to the reader...

It is nice to have a self-contained example that you can run on your machine, but I think an example that actually shows the capabilities of the framework will serve you better. E.g. show an example of serving network requests by running services on multiple nodes, rather than a toy function with a toy database.

Showing that it's as easy as `run_in_executor()` is nice, but not if it does the exact same thing as `run_in_executor()`.

>Showing that it's as easy as `run_in_executor()` is nice, but not if it does the exact same thing as `run_in_executor()`.

That's fair and a good point! I suppose reading this, you have no evidence that anything but local_thread exists. I'll change that, somehow, to show that indeed there exist values other than local_thread, and that passing those will run things on remote hosts.

This article originally grew out of a tutorial on writing tests using this style, for which running over multiple hosts is not necessary and maybe undesirable... but that's not good for what the article is today.

Reply on news.ycombinator.com