The Fossil Sync Protocol | Hacker News Reader

The Fossil Sync Protocol

fossil-scm.org

127 pointsthunderbong44 comments

Dear Jetbrains, if you read this, please reanimate your Fossil plugin. I tried to do it myself, but the learning curve for IJ plugin development is too steep.

I readily admit I am not familiar enough with fossil to know about the impedance mismatch, but I'll point out that https://github.com/JetBrains/intellij-plugins/tree/idea/241.... https://github.com/JetBrains/intellij-community/tree/idea/24... https://github.com/JetBrains/intellij-community/tree/idea/24... https://github.com/JetBrains/intellij-community/tree/idea/24... may a long way toward finding how they think about those operations

and you may already know about it but https://github.com/irengrig/fossil4idea (Apache 2) is the one listed in https://plugins.jetbrains.com/plugin/7479-fossil-integration although as you pointed out, it's very stale

Thanks for the code pointers! Yes, fossil4idea was the plugin I tried to update. I might give it another shot at some point.

Fossil is really nice. I still have some projects (from the pre-Dropbox days) that use it--and that are now archived on it, with 100% fidelity.

Having a single file repository is just amazing.

I use fossil in the post-Dropbox era by keeping my singular repository file inside Dropbox and my working copies outside of Dropbox. Works great for me. Works infinitely better than a git or mercurial repository where the .git/.hg folders get corrupted when there are file conflicts.

Just set up a git server somewhere. It's not hard. Way better than using Dropbox too.

Why would I do that when Fossil and Dropbox already works for me :)

Sidenote: I use git at work and I absolutely despise it.

Because syncing a monolithic binary file is trickier and more intensive than syncing a bunch of little files, once your repo is big enough. If your git repo was corrupted, you might recover some data. If you corrupt a binary database, good luck with that. Dropbox is also harvesting your data for various purposes. I hope your code isn't valuable, because it could be leaked or stolen by employees of Dropbox. There's also something to be said about using what other people use instead of bizarre niche tools.

Disclaimer: I have used Fossil and it's a decent system for small applications where everyone is on board. But I would not recommend it due to the reasons above.

> Dropbox is also harvesting your data for various purposes. I hope your code isn't valuable, because it could be leaked or stolen by employees of Dropbox.

I always thought the encrypted repositories was a neat feature but never really understood what a use case for it would be. I guess this might be one. https://www.fossil-scm.org/home/doc/trunk/www/encryptedrepos...

> I hope your code isn't valuable, because it could be leaked or stolen by employees of Dropbox.

So... you only host your own repos on your own machines, without any kind of EULA? You do realise that hosting git _anywhere_ requires a degree of trust, right?

The fewer people you have to trust, the better. Personally I would not let anyone else host important code for me. I don't own a business but if I did, I would either host it myself or put up tons of roadblocks for anyone trying to steal my code. It might not be impossible but it would be impractical.

Dropbox has had data breaches before. You have to run their spyware to use it too. This is not a theoretical concern.

The problem is that Dropbox specifically has shown themselves to be unworthy of trust. Remember just last year when they opted everyone into sharing data for AI training purposes without asking? A company that does that is not one I trust with my confidential business information.

I think this is the first time I've seen someone say they outright hate git. Why?

I'm not who you replied to, but I think git is fundamentally insane. It's sufficiently opinionated about how it should be used that if you violate certain customs, it can decide to delete commits that you explicitly created. Despite that, it's not opinionated enough to avoid having a ton of subtly different (or effectively the same) ways of doing a thing on the cli. And the cli is an absolute mess of footguns and inconsistent flags.

That's interesting. I've been using git every day for about 8 years, I've never found myself in a situation where I couldn't fix something I broke. But maybe that time will come.

The usual reasons people hate git are directly resulting from that them not taking time to learn it. If you do know how to use it, you will find it difficult to lose any commits. I suppose the way "reset" has multiple and very different uses is a thorny issue for noobs. But if you just read up when you don't remember, you'll be very safe.

I've had decent results with git in Dropbox, and now in SyncThing. There is nothing that git reset --hard can't fix as long as you keep committing to a server...

A git repo in Syncthing? Wow. You really like to live dangerously.

Have been doing this kind of thing for over a decade to sync between my desktop and laptop(s). No serious data loss, ever.

For what it's worth: I've had excellent success with moving the .git folder outside of dropbox. Simply create a text file, ``.git``, in the folder you want to use, and add a single line: ``gitdir: /abs/path/to/gitdir``. Now you can use your sync provider of choice to shuffle most-recent code between computers as needed, without extra git commits.

The biggest gotcha is that it's not particularly cross-platform-safe, because the file paths might be different.

The problem with single file formats is they don't play nicely with file sync tools in what I'd call "common" scenarios.

Due to the way it's laid out, a git bare repository will work quite well with a per-file syncing system because references are stored as files pointing at the hashed object store.

So the worst you can do even without central coordination is have a reference conflict. I'm sure bad things can happen, but for the purposes of "keeping repos synced via syncthing" it works well enough (note: this is different to trying to keep a working repository synced, which I wouldn't recommend).

Since it uses SQLite under the hood, Fossil as a version control system might be able to scale much more gracefully than git for huge monorepos. I think about this often but haven't found a way to effectively use this knowledge yet, since I haven't worked in a giant monorepo shop yet, nor have I found any third parties talking about their experience with this.

For a large monorepo network filesystem approach is inevitable. Every local storage approach has limits and breaks down eventually.

OpenBSD tried to import their CVS tree a while ago, it didn't go well. Now some OpenBSD devs are essentially re-implementing Git as their hopeful long-term solution to getting off CVS at some point.

I love the concept of Fossil being in SQLite, but there's a reason that Mercurial invented revlogs and Git tries to keep related objects close to each other in packfiles. Sometimes, you really do need a dedicated file format optimized for specific use cases. I'm completely unsurprised OpenBSD wasn't able to pull this off.

(Kiln split the difference by storing metadata in SQL Server, but keeping all the actual source data in their native formats. This works great, but is only really viable if you can guarantee things never get out of sync, which is basically impossible for random local Git repos.)

Any mailing list threads in this topic.

Was able to find any and would be curious to read more.

I think it's called Game of Trees (GoT): https://gameoftrees.org/

I found this for NetBSD from 2011 https://2011.eurobsdcon.org/papers/sonnenberger/fossilizing....

TLDR imported cvs2fossil, then fossil to git and mercurial. fossil was less optimized at scale. No dvcs was perfect. At scale things don't always import easily

"The number of artificial commits and conversion glitches could be minimized by cleaning up various issues. This was made easy by exploiting the database and writing small Python scripts calling “rcs” and related programs. Various tests show that the Fossil repository and CVS give exactly the same output for individual revisions. Differences when comparing working copies of specific branches are accounted for.

The performance of Fossil is competitive. Some areas like “fossil pull” need further work, but the majority of the work can shift to improving the user interface."

Many years ago I was using fossil for OpenBSD development to manage my patches.

Around that time I tried to import the entire OpenBSD src repository into fossil, by importing the CVS-to-git conversion of src, as published on Github. I was following the official git->fossil migration guide. I left this running for a week (or two?) at which point the fossil git loader was loading OpenBSD commits from somewhere around the 2000s. At that point I stopped the process. Performance might be better today, I don't know. And perhaps post-conversion run-time performance is much better, but I never got that far. Anyone can try to reproduce these results by running the same conversion today.

I don't think I ever talked about my attempts with fossil to anyone at the time. But I recall the topic coming up somewhere when the Game of Trees project became public, and someone suggested I should be using fossil instead.

I am now using Game of Trees for all my OpenBSD development work and I am happy with it.

They tried using Fossil for netbsd (and may still use it for some things, I'm not sure). Fossil was not designed for large repositories: https://fossil-scm.org/forum/forumpost/5ad122663cbae98f?t=h It could be optimized for that case, but since there doesn't seem to be a lot of interest, I don't know that anyone is going to do the work.

Microsoft uses git for Windows repository pretty successfully. It started with custom VFS implementation and fork of git but by now everthing is upstreamed and VFS is not used anymore.

https://blog.gitbutler.com/git-tips-3-really-large-repositor...

Doesn’t that speak to parent’s point? Git original implementation did not scale to huge sizes. Needed specific development work to change assumptions. Presumably similar design improvements could change fossil.

Python and obviously the Linux kernel are large git monorepos.

SQLite uses Fossil because they want to not only eat their own dog food but they want a complete app that can do most of the work that GitHub can do.

> Each artifact is identified by a cryptographic hash of its content, expressed as a lower-case hexadecimal string [...] It is assumed that the hash names of artifacts are unique - that every artifact has a different hash

Isn't that kinda scary with the birthday problem paradox for people who use Fossil to manage a large number of artifacts

I don't think it will be a problem: <https://fossil-scm.org/home/doc/trunk/www/hashpolicy.wiki>

As a rough approximation, you need something like sqrt(N) entities before the birthday paradox becomes an issue.

The Wikipedia article at https://en.wikipedia.org/wiki/Birthday_problem#Square_approx... gives the approximation as:

  n = sqrt(2 * m * p(n))

where p(n) is the probability of having a collision.

Let p(n) be 1e-9 (a 1 in a billion chance).

With fossil's old default of SHA1's 160 bits you'll need roughly sqrt(2 * 2^80 * 1e-9) = 49 million artifacts before the likely duplicate.

A 1 in a billion chance might still be scary (if everyone on Earth has their own 50 million repo, each with unique artifacts, then there will be duplicates!), but SHA1 use in fossil is mostly a historical issue.

With fossil's new default (since v2.10 in 2019) of SHA3-256, you'll need something like 1.5e34 artifacts before there is a 1-to-a-billion chance of a duplicate.

As MathMonkeyMan's link points out, you can configure your repository to "shun-sha1" and only allow sha3.

It's not nearly that bad. You effectively did the sqrt twice by setting m = 2^80 when it should be 2^160.

You need 5.4e19 artifacts before the chance is 1 in a billion.

Thanks for the correction!

That's exactly the reply I was looking for, thanks

Note that exactly the same constraints exist for Git.

> The "bag of artifacts" data model used by Fossil is apparently an implementation of a particular Conflict-Free Replicated Datatype (CRDT) called a "G-Set" or "Grow-only Set".

Isn’t this just by virtue of content addressing objects? In that sense Git’s odb is also a G-set.

If you look at the definition of CRDTS, it seems trivial that all distributed scms must satisfy CRDT properties. Some observations are trivial once you think about it, but worth saying to connect to different audiences with different interests or backgrounds. I wouldn't be surprised if they added this because they got the question many times. The rest of the paragraph makes clear that they did not develop fossil in the context of CRDTs, and nothing in the rest of the document depends on this observation.

This is beyond me to understand properly, so stupid question time:

Is this, or could this be modified to be, able to sync an sqlite database (or really the data it contains) between devices?

I'm aware of other ways to do this but if there was an internal sqlite way to do this...

No, this protocol doesn't deal with SQLite. In fact, SQLite in Fossil is just an implementation detail, the design doesn't depend on it.

Reply on news.ycombinator.com