Hacker News Reader

Comment by mbreese | Hacker News Reader

This is a clever use of fuse/sshfs and chroots to allow you to run programs on remote computers. However, for many workloads, isn't the network I/O going to outweigh the benefits of remote CPU cycles?

I really think this is a clever hack, but I'm not sure how useful it is. Is it really good for the transcoding example? I could see that being good if the remote machine has a better GPU, but I'm struggling to see how this could work for other use-cases. Maybe compiling (which was one of the main use-cases behind Sun's SGE)?

I do most of my work on HPC clusters, so moving compute out to remote nodes is something I'm very interested in... and I'd love to learn more about what kinds of workflows the authors had in mind.

View on HN

I can totally imagine wanting to work on a thin client on your home network over a 1gbps network connection. Hell, if you optimized specifically for this and ran a 10gbps line right from your thin client to the server, it would be blazing fast.

Another application I could see is putting a loud server into a closet somewhere and just running network cables to it. Make a 4 GPU render node, then just send jobs to it over your thin MacBook.

The thing is you need a client fat enough for normal use but too thin to run the command in question.

For the workflow you describe, why not just ssh or RDP into the beefy machine from the thin client? Wouldn't that make more sense?

Well for starters, no scp command at the beginning and end of each job. I'm sure you could work out an alternative solution, this is just one such solution.

Yeah, this feels like a good solution for when you only occasionally need to get the remote machine to do the work, e.g. the macbook example for simple dev work. In that case it probably is more convenient.

If you were relying heavily on the remote machine, maybe using a pinebook pro or something, ssh or rdp might make more sense.

It's cool to have more options though, I have considered workflows like this before especially in the context of replacing my laptop with something like a pinebook.

Per the readme, architectures aren't cross-compatible, so you can't run commands from an x86 box on an ARM box (and presumably vice-versa). That would rule out the pinebook and most SBCs as potential use-cases.

The thin machine could be your laptop where you'd occasionally want to do a heavy operation on a workstation or server on the data you happen to work on on your laptop. Sounds like a decent usecase.

Of course running it there will work too but will need more setup. So the occasional thing is what really makes this worthwhile.

It's a bit of a niche but I see the benefit.

ssh and RDP doesn't easily extend to live workload migration or other forms of location transparency.

The said method doesn't support cross-platform executions (at least not without some qemu hackery).

I can totally see how I would use this to transcode videos from my NAS using my main workstation CPU since both are connected through a 10gbs switch though (need to figure out how to integrate this with Plex transcoding flow).

I believe you could get a similar result using Jellyfin and rffmpeg [0]. From a quick glance it seems like it should work from ARM->x86

https://github.com/joshuaboniface/rffmpeg

A 4k 60hz 30bpc display requires 14Gbps to feed. The thin client thing is such a meme, all the computing requirements continuously scale - there is nothing "seamless" about them.

A ton of tasks don't require any graphical data streaming, in fact aside from gaming most resource intensive applications can be run in the terminal and thus easily offloaded to another machine. Text editors are already getting distributed code checkers via LSP, it wouldn't surprise me if in a couple years we could run the whole backend of IDEs on another computer and the local part can be more lightweight than most websites.

Then you only need to drive a 4k screen for displaying a desktop and maybe video playback, which can already be achieved with a passively cooled Raspberry Pi.

> it wouldn't surprise me if in a couple years we could run the whole backend of IDEs on another computer and the local part can be more lightweight than most websites.

Checkout the official SSH/Docker/WSL Remote Development extensions to VSCode

Not surprisingly, emacs has been able to transparently do this for a long time, now.

Granted, this all falls flat when your network connection is garbage. :(

Also could use some more polish around handling SSH connections; I often manage to lock up TRAMP when I step away from the computer for 5 minutes. It's something that can probably be fixed with the right ssh_config/sshd_config, but I can't for the life of me figure out the right incantations.

For now, I just habitually do M-x tramp-cleanup-all-connections whenever I stop actively working with remote files, and let Emacs reconnect automatically when I save a remote file or refresh a remote folder listing.

(Incidentally, half of my use of TRAMP went away once I managed to configure a 256-color terminal on my sidearm; with that, Emacs in terminal mode looks almost indistinguishable from a GUI session.)

> It's something that can probably be fixed with the right ssh_config/sshd_config, but I can't for the life of me figure out the right incantations.

Have you tried something like this? It does a common TCP/IP socket for each remote host and does application-level pings every 4 minutes to keep the socket active. Solved most of my "stay connected" issues.

    Host *
      ControlPath ~/.ssh/cm-%r@%h:%p
      ControlMaster auto
      ControlPersist 10s
      ServerAliveInterval 240

For me, I'm just fighting rural networking. Ever since I setup control master for my ssh, I have not had too much issue with the ssh.

I still like running emacs locally.

"I've finally got my emacs setup just how I like it. But my gigabit internet connection isn't fast enough for a text editor these days..."

If only I had gigabit... 4g lte that constantly drops is not fun.

Meanwhile I have a 4G modem at home because it came with higher stability and more than double the bandwidth compared to a similarly priced cable connection. Sometimes the more traditional telecoms are working really hard at pulling an Intel.

To be sure, this is my best option at the moment. Can't get any cable connection.

I wrote the "more lightweight than most websites" part for a reason. VS Code is already horribly sluggish on my Thinkpad with i7 CPU, I would go totally insane trying to run it on anything referred to as thin client. Might as well go for a full-blown IDE at that point, some of them are actually trying to optimise for low-latency text input.

> if in a couple years we could run the whole backend of IDEs on another computer

It’s been done. Look at AWS Cloud 9, or Eclipse Che.

I think if you were to send the raw frames sure, but wouldn't they just send the delta of the frames (only areas which changed) -- with some sort of compression algorithm and parameters like VNC does?

SPICE solved this (years ago in fact) by doing various sorts of compression, including detection of video, and streaming that over the network. (https://www.spice-space.org/features.html)

I agree that computing requirements scale as we go, but surely you don't need 4k at 60Hz to browse the web now? I regularly play games using steam link now and while it's not the full quality and framerate, it's just convenient (and cheaper than having a full desktop pc in every room). Thin computing seems much more seamless and workable now than it did in my college years with Sun thin clients.

For a typical HPC cluster, there is a shared filesystem with exactly the same file structures across the cluster over a high bandwidth network, which basically replaces "a clever use of fuse/sshfs and chroots" with something far better. OP's project is clearly useless in HPC setting.

On the other hand, OP's project is exactly an linux/posix version of the same as what Plan 9 was designed to do and has been doing.

I miss openMosix[1]... Its been so long, but I remember running kernel compiles on two workstations a long time ago... Double the core count! It actually gave a significant speedup on 100mbps switched lan (I think it was just a crossed cable, no actual switch).

I couldn't remember setting up NFS or anything like that - apparently openMosix did some tricks too: http://www.firewall.cx/linux-knowledgebase-tutorials/openmos...

[1] https://en.m.wikipedia.org/wiki/OpenMosix

> project is clearly useless in HPC setting

Which is why I'm curious about the applications they had in mind. I know that my workflow isn't typical for most people, so hopefully others have better uses in mind.

For me, I just submit adhoc jobs to SLURM and call it a day. Not everyone has access to an HPC cluster or is comfortable setting such a system up for a small, home cluster.

Anything (like this project) that makes HPC-like processing more accessible, I'm interested in.

When I developed this I was very much thinking of using this to run one-off compute heavy tasks, e.g. compressing a video, running scientific computations or rendering something in Blender. You would just rent a VM for a few hours with the hardware you need, set it up in a few minutes, run your job there and delete it again.

OP’s project is actually a good idea at development stage, where data is not the main thing yet..

It reminds me of GNU Parallel[0], so I have to imagine it fits for some use cases. I stumbled upon Parallel when I was trying to do user-level "distributed" computing at the day job for running hundreds of simulations in a shorter amount of time. Never really did get it working (I had a hard enough time just ginning up a Python program to orchestrate test cases across local cores; still took 3.5 hours for a full run of tests), and they're currently trying to get a SLURM cluster setup while I'm debugging NVidia binary drivers and overloading our Gitlab instance.

[0] - https://www.gnu.org/software/parallel/

Compilation is a good example of where using remote machines can help substantially: https://docs.bazel.build/versions/master/remote-execution.ht...

> for many workloads, isn't the network I/O going to outweigh the benefits of remote CPU cycles?

Definitely, but, there are also many workloads (and situations) where it doesn't and this works great.

One that comes to mind is intense build jobs.

Sometimes I like to transcode videos I recorded on my windows machine during a gaming session to post to our group chat.

If I have to do those steps locally, I can't be gaming. That's not cool.

Still, kinda makes more sense to just fuse/smb the videos and have the remote processing node pull, processes, push.

> isn't the network I/O going to outweigh the benefits of remote CPU cycles?

Not if:

   a) your network B/W is higher than your local disks (not as unusual as you might think, spinning medium is awful slow)

   b) the app you're using chunks up data in a reasonably efficient manner so as to not get bitten by latency issues (ffmpeg encoding likely meets that bar).

> for many workloads, isn't the network I/O going to outweigh the benefits of remote CPU cycles?

Yes, the README addresses this:

> File system performance remains a bottleneck, so the most suitable workloads are computationally constrained tasks like ray tracing and video encoding. Using outrun for something like git status works, but is not recommended.

> Maybe compiling (which was one of the main use-cases behind Sun's SGE)?

The problem is that your task will still be limited by power of a single machine.

You need means to run a task across more than one server, and preferably great many.

It's possible, but completely impractical to run a shared memory system across network.

If somebody figures out how to do that usable in practice, and without complete rewrite of the program that will be really impressive, and an ACM medal material.

It's not trivial, but it's certainly possible. <Un-named IC customer> has been doing level 1 image formation processing for decades using shared memory across a network. They're even doing it in Kubernetes now utilizing libraries from Sandia National Laboratories that use a volume mount of an in-memory filesystem to running containers so you can scale your compute cluster up and down as your memory load varies. This basic setup is powering the entire US geointelligence enterprise. They call it the Tumbo Framework. There's a press release alluding to it here: https://www.sandia.gov/news/publications/labnews/articles/20...

ScaleMP, SGI UV, and HP Superdome all disagree with you. Arguably, plain old RDMA qualifies under this phrasing.

I took the parent comment as meaning on a regular Ethernet+IP network. SGI UV and HP Superdome are semi-exotic hardware architectures. ScaleMP requires Infiniband, which is at least 10 times, if not 100, the price of Ethernet; it also requires fiber optic cable for runs > 5 meters.

I should add to my above comment about the IC doing this kind of thing in Kubernetes on EC2 instances that they utilize the Amazon Elastic Fabric Adapter, which bypasses the ethernet NIC to avoid all of the queueing delay you can't tolerate in applications that need to be basically unaware that some threads are local and some are remote. And obviously they make sure your servers are in the same or physically nearby racks.

I thought modern variations of Ethernet (Gbps+ range) were competing with traditional uses of Infiniband nowadays.

UV and Superdome are custom hardware for huge NUMA boxes,so not a great comparison. ScaleMP is definitely valid though - a real shame it is stuck behind a license fee, would be interesting to experiment with.

Even a hardware RDMA is snail slow for launching a non-adapted piece of software over a network on shared memory.

HPC programs which do use shared memory, were written with network latency in mind from the start.

I regularly build LibreOffice. It would be great to use a faster computer.

Edit: yeah, I posted too fast. I am not their use-case.

Somehow I think this may make it possible to "horizontally Scale" the desktop experience. What if I have my main computer/cpu at home and have additional headless computers configured and I set my Linux system so that everything besides the Kernel/boot stuff is spread through the other computers?

Reply on news.ycombinator.com