Fast file synchronization and network forwarding for remote development
github.com
github.com
[0]: https://mutagen.io/ [1]: https://mutagen.io/documentation/docker-desktop-extension [2]: https://mutagen.io/documentation/orchestration/compose
[0]: https://github.com/mutagen-io/mutagen/graphs/contributors
Thanks for this great tool.
there are issues occasionally with symlinks (especially with large node_modules folders) but most of the times nothing breaks and they are easily fixable - running mutagen sync monitor shows you things as they happen. one thing to consider is where .git directory is hosted. I personally keep git on windows side - you should make sure it only exists on one side and add an exclusion (mutagen has a parameter for that). performance wise, on SSD, an npm install from zero takes at most 5 seconds to sync (running npm on wsl2 -> files appearing on IDE).
Do you want something different from the "one-way-replica" mode?
I understand that this is probably an oversimplification, but I’ve read at least two separate stories that are along those lines. The comments for both stories were basically “yeah that sounds right because of the way you set it up”. It’s not a bug so much as how synchthing works. My understanding is that there is a circumstance where you sync an empty folder with a folder with your documents, and it decides to synch the empty folder such that it erases the documents. Without any kind of warning. Wild.
it is possible to use syncthing for backups, but that is actually one of the pain points. there is an option to specify to ignore deletes, but that option is hidden in advanced settings, and there is no quick way to check if it is turned on. every time i delete something that i want to keep on the backup side. (like pictures from my phone, where i am running out of space) i first go to the destination/backup server and double check that this option is turned on to make sure that i won't accidentally have the backup deleted too.
it's also an issue that i can't see the remote settings from the client side. i have to connect to the remote server to check.
there is also a problem with the way it resolves conflicts. if you have "ignore delete" set up, and then you delete files, syncthing will tell you that the two locations are out of sync. it offers a button to force a sync, but there is no preview explaining what that button will do. there is also no choice in how the out-of-sync status should be resolved (in which direction the sync should happen to get the two locations in sync).
i have always been afraid to ever push that button, and there is no way to make it go away. i think this is an UX bug, because if i intentionally configure to ignore deletes, then deletes should not count as a difference. they should actually be ignored.
It's also nice for automatically managing port forwards.
That said, we have developers using VS Code, JetBrains, as well as Vim. The Mutagen-based workflow works well with all of them.
Plus, it's multi platform. I'm using it to synchronize directories between hosts running macOS, OpenBSD and Linux. Everything works fine.
I haven't tried the Docker Desktop extension since I switched to Colima (Docker Desktop is constantly broken on Apple Silicon).
Has anyone tried this for a real-world project and can share feedback?
Are you able to provide a reference to how Mutagen secures my code on an untrusted remote?
So, for example, Mutagen doesn't implement any encryption, instead relying on transports like OpenSSH to provide the underlying transport encryption. In the Docker case, Mutagen does rely on the user securing the Docker transport if using TCP, but works to make this clear in the docs, and Mutagen is generally using the Docker Unix Domain Socket transport anyway. When communicating with itself, Mutagen also only uses secure Unix Domain Sockets and Windows Named Pipes.
When it comes to permissions, Mutagen doesn't do a blanket transfer of file ownership and permissions. Ownership defaults to the user under which the mutagen-agent binary is operating and permissions default to 0700/0600. The only permission bits that Mutagen transfers are executability bits, and only to entities with a corresponding read bit set. The idea is that synchronizing files to a remote, multi-user system shouldn't automatically expose your files to everyone on that system. These settings can be tweaked, of course, and in certain cases (specifically the Docker Desktop extension), broader permissions are used by default to emulate the behavior of the existing virtual filesystems that Mutagen is replacing.
For files at rest on the remote, I guess I assumed files would be encryted on the remote with a local key since GP said "one can develop on untrusted remote machine" and "VSCode remote always assumes that the remote part is trusted".
On an actually untrusted remote, removing group read permissions doesn't do much to secure my code.
The only scenario where it's helpful is a system with multiple non-admin users, perhaps like a university lab computer but who's doing sensitive work on those anyway?
Shared systems with multiple non-admin users was one of the original motivating use cases for tighter default permissions.
I don't think there's any scenario where one can perform truly secure development work on an untrusted system. You could certainly store encrypted code in an untrusted location, but there's not much you could do with it on that system (without a hypothetical compiler or tool that maybe supported some sort of homomorphic-encryption compilation operations?). Even decryption on-the-fly for processing by regular tools wouldn't be secure on an untrusted system. And running any code there would be equally insecure.
I'd imagine that for any seriously sensitive work, one would only want to work in highly controlled, trusted, and firewalled environments. If there's a scenario I'm missing though, definitely let me know.
Just to be clear, when I mention sensitive work, I'm not necessarily talking about national security, military, etc. kind of work. Any work for a client is sensitive enough that I wouldn't do it on any remote I (or my client assuming there is approval) don't control.
I will try Mutagen at some point. The fact that it's editor agnostic is certainly a big sell!
- Mutagen tries to integrate recursive filesystem watching very tightly into its synchronization loop to drive synchronization and allow for near-instant filesystem rescans
- Mutagen automatically copies an "agent" binary to remote systems to support synchronization, so no remote install is required
- Mutagen uses Protocol Buffers for its data storage, so synchronization sessions created with older versions continue to work with newer versions
- Mutagen written in Go, Unison in OCaml (which allows Mutagen broader platform support "for free")
- Mutagen tries to treat Windows as a first-class citizen
- Mutagen uses race-free traversal (e.g. openat, fstatat, unlinkat, etc.) to perform operations
Obviously the internal implementations are different, but both use differential (rsync-style) file transfers, both use the same reconciliation concepts, etc.
Mutagen has the advantage of Go, recursive filesystem watching, and modern POSIX/Windows APIs that didn't exist when Unison was originally written, though some of that functionality has been brought into Unison.
For a comparison with Syncthing (and to some extent Unison), check out this comment[0].
- Mutagen performs bidirectional synchronization (though it can also operate unidirectionally); rsync is unidirectional
- Mutagen uses recursive filesystem watching to avoid full filesystem rescans (whereas rsync always does a full filesystem rescan). This allows Mutagen to provide a more "real time" sync.
- Mutagen has an active synchronization loop that doesn't require manual invocation.
- Mutagen has more idiomatic Windows support.
- Mutagen doesn't require that it be pre-installed on both endpoints.
Both use differential transfers (i.e. the "rsync algorithm") for transferring individual files.
There are other differences, of course, as well as similarities. Mutagen's design is tuned for development work, rsync's design is tuned for replication. I still use rsync for archival operations on a daily basis - it's great!
Does Mutagen handle the case where “local tools” (running on a completely different architecture than the remote) still need to “know” about include/header/library/etc. files from the remote machine in order to provide working “intelligence” capabilities?
It’s one thing to efficiently sync “code”, but it’s another to make local tools fully-aware of the remote system’s header files, libraries, etc.
Mutagen will, however, happily operate between different operating systems and architectures, so things like working with a remote amd64-based Docker engine from your local arm64-based laptop are totally possible.
Also, several external projects (such as DDEV[0] and Garden[1]) do use Mutagen as a low-level component in their stack to provide synchronization that does "know" a bit more about the framework that you're using.
[0]: https://ddev.com/ [1]: https://garden.io/
To avoid a large class of race conditions (at least to the extent possible allowed by POSIX and Windows), Mutagen will use `*at` style system calls for all filesystem traversal on POSIX systems, with a similar strategy on Windows.
Also, to avoid race conditions due to filesystem changes between scan time and change-application time, Mutagen will perform just-in-time checks that filesystem contents haven't changed from what was fed into the reconciliation algorithm.
[0]: https://mutagen.io/documentation/synchronization#modes [1]: https://github.com/mutagen-io/mutagen/blob/master/pkg/synchr...
[0]: https://www.cis.upenn.edu/~bcpierce/papers/index.shtml#Synch... [1]: https://www.cis.upenn.edu/~bcpierce/unison/
Reminds me of my occasional use of rsync. I'm always afraid of invoking it wrong and synching in the wrong direction.
unison has a UI to manage conflicts on a file level, whereas synchthing only allows a manual overwrite of all changes at once, and i have to log into the remote daemon if i want to overwrite changes there instead of locally.
syncthing also will not tell me if all local changes arrived at the remote side, or if the remote is still busy downloading, without logging into the remote daemon
on the other hand, unison only manages one source/target per process if i remember correctly while syncthing can manage many different source/target pairs conveniently.
how does mutagen compare in these two areas?
Mutagen provides fairly detailed reporting of synchronization status, file staging progress, and change application problems via its "mutagen sync monitor" and "mutagen sync list" commands. These also support JSON output (and Go-template-style formatting), so you can pipe this information into other tooling. If you need REALLY detailed information, you can look at the debug or trace-level logs from the Mutagen daemon, but that's typically only for debugging during development of Mutagen itself.
Mutagen is somewhere in between Unison and Syncthing on the topology front, but closer to Unison. It still only supports two endpoints per synchronization session, but unlike Unison it doesn't require that one is local (i.e. you can do remote-to-remote sync using your local system as a proxy). But with both Mutagen (and Unison), you can set up a hub-and-spoke topology if you want to sync with multiple nodes.
Two features I didn't see addressed:
- Is there a GUI of any kind? I use Syncthing's web GUI + a tray applet for monitoring syncing on desktop, and on mobile there's an app for syncing also. Makes it easy to detect if there's a problem with any of the folders being synced, and easy to add new folders this way also.
- Is there an equivalent to Syncthing's untrusted devices feature? It basically allows for client-side encrypting the files, so you can have say one remote with the encrypted files and another remote with unencrypted files.
All of your readdir()/stat()/open()/read()-style calls will suffer significantly on virtual filesystems, and unfortunately these get hit a lot by things like IDEs (e.g. when indexing code), compilers, and dynamic language runtimes (especially PHP).
No tool is at fault in this chain, of course, it's a hard problem. Mutagen is able to offer better performance by being a little less dynamic and creating "real" copies of all the files on a more persistent filesystem.