Emacs' Org-Mode and Syncthing = Perfect (2017) | Hacker News Reader

Emacs' Org-Mode and Syncthing = Perfect (2017)

hiepph.github.io

135 pointsRMPR77 comments

The amount of times I have managed to slightly misconfigure Syncthing and as a result it _deletes all my files_ is incredible.

Last time, I configured a Receive only folder in device Y, but forgot to mark it as "shared to device Y" in device X's settings.

Net result: directory erased in device Y.

Sure, I still had the copy in device X, but that was gigabytes that needed copying around, a couple hours of wasted time. It's just mind boggling that Syncthing would happily go around deleting thousands of files without warning specially with a minor config error .

> The amount of times I have managed to slightly misconfigure Syncthing and as a result it _deletes all my files_ is incredible.

That only happened to me once (when I was initially setting up Syncthing), but even once is too often for something like this.

The incredible part for me is that there was no clear indication in the UI that I was doing something destructive, I only realized that Syncthing was destroying my files because I was manually keeping an eye on it through my file browser. Surely deleting local files should require the user to first confirm using large warnings and red buttons?!

I double that. Some years ago I almost lost the entire synced state after adding a new device that hadn't synced anything yet. For some reason syncthing decided that the new device has the most recent version of the data and happily started deleting everything on the other peers. Thankfully I had a backup, and not all peers were connected. After that I concluded that syncthing is fundamentally broken.

I sync my org files using a low-tech solution: I have a .dir-locals file that injects a post-save hook that automatically commits changes to git and pushes them to the remote. It's stupid, but it works, and I never have to think about it.

I initially misread "I double that" as "I doubt that" and was confused by the rest of this comment

Maybe it is because I always do 2-way sync folders, but I've used Syncthing for over 2 years now without ever having a hiccup.

I do take regular backups to Blu Ray optical disks 'just in case' but have yet to have need of them.

I thought about doing that too, but BluRay discs are kinda small these days.

Ah yes, the typical problem of Open Source applications, especially ones without a company behind it.

The implementation of the actual sync is pretty good (I never had a problem with it).

But, the UI is not intuitive at all. Its hard to setup a new sync, and if you're not careful you can set it up in a way that deletes all your files without any warning.

While I've lost data to Syncthing too, I've also lost data by removing a Dropbox folder. I don't think bad UI is the sole problem of FOSS.

There are enough company-backed products with terrible UI to make your argument invalid.

Software UI is hard, that's it.

Absurd.

If anything, it is the UI of Synctying that allows me (sometimes) to try and protect some directories.

When the same thing happens with Dropbox, all my computers automatically lose all files because there is no way to force a one directional sync to one specific directory.

This did happen for me one time in the past, which turned me off Syncthing then. Since that time though, I've tried again and never had this happen. I assumed it was an edge case of some sort that they fixed, but you're saying you've had this experience recently? (As in the last year or two?)

As far as I know it was fixed, but yes it was last year or two (the receive only setting is very new). I had similar issues in the past, though.

This is why I use Git for this (running MGit on my Android phone). It's really not a huge hassle to do this manually once a day or so (maybe more if I'm doing some out of band stuff).

When doing this, I was always wondering how to name my commits, when I was in a good mood I wrote good and meaningful commit messages, but when it was not the case, I just slap "aaa". In any case, nothing prevents you to use both, syncthing for a more reactive sync and git as a kind of backup.

For "running" files I've rarely found commit messages to be all that useful and instead just like to diff head versus last week or something.

I have a git alias setup called "git autopush" that adds all, commits with a generic "[autopush]" message, and pushes to the upstream remote. I use this for my repos of "running" files like org files etc. I've modified it recently to require a config param to explicitly enable it because I'm afraid of typing it in work repos :)

alias gl='git add *; git commit -am"lazy"; git push'

To avoid sync issues it's important that Emacs is set up for both autosaving and reloading on external changes.

If you add Orgzly to the mix for mobile access, you also want to configure Orgzly to auto-save whenever you open it or make changes.

With these two adjustments this becomes a pretty good setup.

Just so it's in the thread: I believe the emacs option for "hey, that file changed on disk; I'll reload" is

(global-auto-revert-mode 1)

I've been using emacs for about 15 years and didn't realize this was an option. I've added it to my config - thank you!

(and thanks to the GP as well, of course!)

I am a total emacs noob given my relatively shallow exploitation of it, and my absolute ignorance of lisp. This is absolutely the first time I've provided an emacs answer someone found useful in a technical forum.

I'm gonna need a minute.

(heh)

To build on this tip, the following will shorten file revert time but a couple seconds (default is 5 secs)

(setq auto-revert-interval 1)

Found in https://www.gnu.org/software/emacs/manual/html_node/emacs/Re...

I still had many problems with writes not making it out of syncthing before the laptop/phone would go to sleep - thus creating lots of conflicts to resolve all the time.

It was a frustrating experience, because basically ~once a day I'd have to manually fix diffs on each side.

You can also set up a git repository as a kind of backup, MGit is the mobile client I use for that.

I love org mode (with evil for vim goodness), but the one problem I really wish was easier is image support. Also sometimes a few missed key strokes can really screw up a document! My normal mistake is hitting "J" instead of "j" (vim command for combining the next two lines vs just navigating down one line).

I am always interested in seeing what else is out there for organizing, glad to see org mode is still popular!

>I love org mode (with evil for vim goodness), but the one problem I really wish was easier is image support.

Yes, inline images are an important part of rich documents. I'm not sure what you're looking for in terms of images, but I've written this function which allows you to paste images directly from the clipboard into documents:

https://paste.debian.net/plain/1144961

I've tried Org+Evil a couple of times and really liked it, but missing key strokes always make me go back to Vim+Markdown. If I mistype something in Vim it's trivial to undo/redo to get where I need to be, but I never managed to get my head around the Emacs undo behaviour and that just scared me away.

If you don’t use undo-tree then emacs undo works like this:

1. You have a linked list of states with your current state:

  A->B->C->D->(E)

2. Pressing undo moves you to earlier states. Once:

  A->B->C->(D)->E

Twice:

  A->B->(C)->D->E

Doing anything that isn’t undoing again will add your undoings to the history:

  A->B->C->D->E->D->(C)

So now when you undo you in some sense redo what you had preciously undone, then undo it again, then undo further.

The advantage is that you never lose history by undoing. The disadvantage is that it can be confusing.

Undo in region complicates this.

Thanks for the explanation, that makes sense. Good to know that states A and B are still accessible, I remember thinking they'd disappeared forever. Next time I try emacs (no doubt there'll be a next time!) I'll bear this in mind.

I'd consider using undo-tree instead. First, it might make more sense. Second, afaik Vim has about the same model (haven't used it in a while though).

Undo-tree's navigation interface isn't a pinnacle of intuitiveness, however.

The tree in vim is weird and hidden. Basically there is an operation to go from a state to the state from which it was derived (“up the tree”) and an operation to go forward/backward in time (of state creation), which is basically navigation in changenum space. Ie if you make some changes:

  A -> B -> C -> D

Undo:

  A -> B

Make some other changes:

  A -> B -> E

Then undo takes you up the ABE chain but if you use g- and g+, you navigate through the list ABCDE

> g- and g+

Those are the operations, yeah! They went a long way for me. I think there's also a command or a third-party addon to see the undo states, in the undo-tree spirit—but surprisingly I've never needed it.

What would you like to see for image support?

An easy way to copy/paste would be good. Right now you need to save a file and type in a link, IIRC

To drag-n-drop an image into an Org document, there is org-download package

I've been a happy Dropbox + org-mode user across multiple devices for some time. I tend to divide up my org files across different subjects and then tie them into org-agenda. I also archive my org items on a regular basis to clean up certain files.

What's the benefit of Syncthing? I know nothing about it.

> What's the benefit of Syncthing?

Not needing a service/server is probably the main benefit, compared to stuff like Dropbox or Nextcloud.

I am also curious.

I took the time to configure Dropbox from the command line on my headless Linode, and while it's not foolproof -- which is at this point probably on me, since I don't have it running as a proper init or whatever -- it's so far Good Enough, and as such I'm not super convinced going to a secondary sync tool is useful to me.

(I mean, since I already rely on Dropbox to manage sync of a broader set of files between 3 other computers + my iOS devices.)

I use it for everything that I don't need to share with other people, because it scales 'for free' with the size of my data (my music library, for instance), and because I can use it across many different devices but selectively sync different 'types' of things to different devices (e.g., a large photo collection doesn't need to be on my mobile phone).

Who can tell me which Android app should be used? Syncthing [1] or Syncthing-Fork [2]?

----------

[1] https://play.google.com/store/apps/details?id=com.nutomic.sy...

[2] https://play.google.com/store/apps/details?id=com.github.cat...

I personally would only use the main one unless I had a specific reason to need the fork. I've never had any problems with Syncthing from F-Droid (and syncthing lite for the wimpy cheap android tablets)

I use Syncthing and it works really well, never heard of Fork though.

EDIT: Fork looks really good too, I've installed it to check.

The fork works way better as an app. The original just wraps the whole web interface as a single screen inside the app, so you have to trod through that web interface to do any setup.

On top of that, afaik the original is rather sparsely developed and released.

I'm using the fork after having issues with the sync stopping and restarting only after device reboots with the original version. So far so good.

Syncthing-Fork is great, allows to adjust sync settings (i.e. network/battery) per-folder

I've been using Emacs org-mode + Syncthing + Orgzly (org-mode on Android) for 2 years now, and it's my favorite notes, todo-list and reminders setup so far.

On top of that, my org folder is under version control, very useful if you somehow manage to lose all your devices (very unlikely but... the Tao of backup you know http://www.taobackup.com/)

Similar to my note setup with one change -- Vimwiki + Syncthing. I'm also using(abusing) git to daily auto-commit everything, so I could delete everything without worry, and browse history with tooling I'm already familiar with. This setup works quite well across all the devices I use (desktop, laptop, phone). With a caveat that phone setup is mostly read-only (apart from ticking the grocery list on occasion) due to it being just a folder with text files there.

I'm actually curious to hear from other vim to emacs org-mode converts since the article didn't elaborate on the cons and pros of the switch. How significantly is org-mode better than vimwiki and is it worth exploring for someone who's been baking in vim ecosystem for a decade?

"Evil Mode: Or, How I Learned to Stop Worrying and Love Emacs" by Aaron Bieber is an interesting talk on switching from vim to emacs + org-mode.

https://www.youtube.com/watch?v=JWD1Fpdd4Pc

Thanks, I just watched the video, and it did intrigue me enough to install spacemacs to start playing around with it.

Having "real" spreadsheet like tables within my wiki does look appealing. I do in fact already have a bunch of small spreadsheets doing some very basic stuff, and it does make more sense for them to be within the wiki itself.

Syncthing is really nice. It does a thing, and does it well. Once it's set up on the devices that share some stuff it just works.

Another combination that works really well for me is the pass Password store¹ and Syncthing to share some passwords with another user.

I have pass set up to have a subdirectory shared with my partner (by registering both our GPG-keys in a .gpg-id file in that directory), and that subdirectory is synched by Syncthing to our devices.

So now we can store shared accounts there, and have them available in our respective password managers.

1: https://www.passwordstore.org/

I agree about Syncthing, I've been using for various folders between various computers and it always works really well. I even have it sync my programs directory. It's fast, very light on resources and syncs very well.

I can't recommend it enough.

Do you mean pass can encrypt with multiple keys so you can share creds? I thought it only supported one gpg per store.

From GP's link:

> Multiple GPG keys can be specified...

No, you can use as many keys as you like. By default it uses the local user's key, but you can specify a list of keys in any directory part of pass' store by adding a .gpg-id text file with one key fingerprint per line.

In particular I use this so that I can have one gpg key for my iphone (which is secured with a long passcode + face lock) and one for my laptop (where I use a gpg key that's baked into a yubikey). I keep it in sync with a git repo (I started with a free tier CodeCommit back before github allowed unlimited free private repos).

I've been using the same combo for a few years. You can run a Syncthing image off Docker Hub directly e.g. using compute engine and optionally persist config and data in volumes, making for a pretty easy setup.

In case anyone is interested in learning more about Org-mode, there are a lot of wonderful tutorials out there. I've found the trick is to start small, usually just with some tasks, and let it grow from there as your needs change.

Some great guides:

https://orgmode.org/worg/org-tutorials/org4beginners.html

https://orgmode.org/orgguide.pdf

An invaluable resource is this Google TechTalk https://www.youtube.com/watch?v=oJTwQvgfgMM it was my first introduction to org-mode.

I’m syncing my org files with Nextcloud, with an S3 bucket configured as a backend [1]. I can even sync to my org files and agenda to my phone configuring beorg to sync to Nextcloud via WebDAV. I’m still pretty new to emacs and org but this is very close to what I would consider an ideal note / task management solution. It’s all plain text (even in S3) and S3 is configured with server side encryption and versioning enabled.

I highly recommend Doom Emacs [2] and Zaiste Programming’s DoomCasts playlist on YouTube [3] to those just getting started.

[1] https://docs.nextcloud.com/server/15/admin_manual/configurat...

[2] https://github.com/hlissner/doom-emacs

[3] https://www.youtube.com/playlist?list=PLhXZp00uXBk4np17N39Wv...

Unfortunately, in my experience using Syncthing while doing cross-platform development (coding on a Mac, compiling and testing on a Windows machine, both machines connected to the same Ethernet switch and sitting right next to each other), syncing with Syncthing is far from instant.

If you just want a cheap Markdown editor with instant syncing, I recommend self-hosted CodiMD. Of course that’s not an option if you actually want the full power of org-mode.

That's odd, it's pretty instant for me. I've been an extremely happy user for years.

Not sure if it has anything to do with file count. I sync a couple thousand files in a directory with close to a million files (aggressively ignored most), although I was talking about changing one file at a time and having the change show up on the other machine maybe 15 to 20 seconds later — or longer, can’t recall exactly. Scanning is very slow too (this is on a fast SSD).

Hmm, that's definitely not typical for me, it only takes a second or two for files to show up. It should be using inotify to be notified of changes, but I don't know if there's a difference on OS X (I'm on Linux).

My experience has been on macOS and Windows, slow both ways. I haven’t used it on Linux.

On macOS FSEvents or kqueue can be used, I’m not sure which one is used. They are definitely using a notification mechanism though, otherwise changes won’t even happen in 15s.

Have you checked "Monitor filesystem for changes" in the config? It polls if you don't do that, and I think it doesn't get checked by default, but I'm not sure.

Yes, "watch for changes" is on for all my synced directories, and on by default.

According to https://docs.syncthing.net/users/syncing.html

> There are two methods how Syncthing detects changes: By regular full scans and by notifications received from the filesystem (“watcher”). By default the watcher is enabled and full scans are done once per hour.

I have the same setup, with one addition -- I use termux as an android share target so I can easily capture files and notes on the go. http://notes.neeasade.net/mobile-capture-with-orgzly-termux-...

Thanks for posting about this. Sharing urls/files/text/etc to termux is something I never thought about doing. I usually end up sharing urls to the gmail app and just sending it to my todo@ e-mail to go through later. I like your idea of sharing to a script in termux better though.

Is your script available somewhere? Is it just a simple "echo org-stuff >> capture.org"?

I am syncing between Orgzly and Emacs by using an automated git script via systemd in laptop and crond inside Termux in phone, along with a git forge for upstream. This is the only setup that worked for me (I don't have a personal server to keep syncing with syncthing and Orgzly WebDav experience was frustrating. )

I wish there was also an iOS version, that I could also access my notes from there.

There is. It’s called Beorg. It’s fantastic.

https://beorgapp.com/

It is indeed, but within the context of this overall discussion, note that BeOrg doesn’t support SyncThing.

There isn’t an alternative for that particular user case without dual-syncing with another sync provider. Best I can tell, there are zero iOS apps of any kind with SyncThing support.

You can (and I have) configure SyncThing to track and sync changes within the iCloud app directory corresponding to BeOrg. It definitely feels janky and I worry about file corruption.

There's no (working) iOS SyncThing app, unfortunately: https://forum.syncthing.net/t/ios-iphone-crowdfunding-on-bou...

Would this help?

https://beorgapp.com/

I'm a refugee from the Great Google Tasks switch to a tiny UX and switching off their Canvas (full screen) mode. Had to migrate thousands of tasks. I used https://tasks-backup.appspot.com/ but that has shut down. Don't know of an easy alternative.

I now use a combination of Orgzly, Dropbox, Tasker and Emacs to keep my desktop and mobile notes in sync and it works great.

The benefits are: plaintext/markdown (org-mode), automatic backup with restore (dropbox), open source or free apart from I think a couple of pounds for Tasker and of course whatever you are paying for dropbox. Plus, unlike Syncthing, only 1 version of a to-do list file at any one time, so less conflicts. Wider benefit of to-do list customisation limited only by your imagination and patience, via emacs lisp.

My various To-do lists (split according to the Eisenhower Matrix) are saved in a dropbox folder.

My Orgzly app is set to read from that dropbox folder: Settings > Sync > Repositories > Connect to dropbox

My Tasker is set up as Tasker > Tasks > New '+' > Enter Name > New '+' > Code > Run Shell

Copy and paste into Command field:

am start-foreground-service --user 0 -a

com.orgzly.intent.action.SYNC_START

com.orgzly/com.orgzly.android.sync.SyncService

You can also set a helpful notification via Alert > Notify > Title

Then in Profiles I set the task to run

- Every hour

- Whenever Orgzly is opened or closed

- Whenever the phone display is switched on or off

Takes about 2 seconds to sync, whenever triggered.

One problem with my device (Huawei) is the very aggressive battery optimisation which simply can't be turned off via their settings. You can fix this by going in via ADB as follows:

adb shell pm uninstall --user 0 com.huawei.powergenie (thanks to https://dontkillmyapp.com/huawei)

You may need to use ./adb (thanks to https://stackoverflow.com/questions/7609270/not-able-to-acce...)

and Settings > System > Dev options > dev options switch on (thanks https://www.xda-developers.com/install-adb-windows-macos-lin... )

... It's crazy that in 2020 I have to set all this up for a smooth experience. What I really, really want is an org-mode compatible (I'd live with regular markdown) web app with offline access and auto-sync, accessible from my mobile device and desktop browser and customisable or just any sane keyboard shortcuts. If free or open-source in Vue.js or React + Firebase even better.

Missed opportunity for https://noteplan.co/

Looks slick but not comparable. That's (1) pay and closed, and (2) apple only.

Reply on news.ycombinator.com