Fossil – Simple, high-reliability, distributed software configuration management | Hacker News Reader

Fossil – Simple, high-reliability, distributed software configuration management

fossil-scm.org

128 pointstypedweb84 comments

I use fossil; like it a lot.

If you use/would like to use fossil, you might also want to take a look at http://chiselapp.com

It's a bit github-ish; online repository hosting, private or public, and free. (I'd happily pay, but they don't provide even a donation page. So: thanks!)

Thanks for the link. I use Fossil for a few projects. While it's not too complicated to set up Fossil repositories on the web, the Chiselapp "hub" would make it even easier and probably a lot more reliable.

One of the things to like about Fossil is its concept of record immutability, that is, data is retained intact in the database even when superseded by a newer version. Fossil asserts a highly ethical position, that revising historical facts is not an acceptable practice.

Using Fossil over the last couple of years, its performance and stability have been rock solid, and I haven't experienced any data loss or other problems with it.

> One of the things to like about Fossil is its concept of record immutability, that is, data is retained intact in the database even when superseded by a newer version. Fossil asserts a highly ethical position, that revising historical facts is not an acceptable practice.

There is big drawback in making it very hard to edit "history": you are bound to carry in your project all sorts of things that you do not want to. And this becomes an even worse problem when you combine source code, tickets and a wiki as Fossil does.

Somebody posts spam as tickets? Now you are bound to have traces of that spam forever in your published repository.

I switched away from bzr to git, just like many others, also because bzr made it impossible to fix small stupid mistakes. OK, I pressed Enter while reaching out for the shift key and made a commit with the wrong commit message. Why do I have to keep that? What does the project gains from this?

I think the problem here is with the word "history". We attach a lot of meaning to that word and nobody wants to be a "history rewriter" or a "revisionist". I prefer more neutral terms like "patch queue" or "version chains". "I polished the chain" sounds much better than "I rewrote history".

I agree that there is a difference between the actual history of a project and the writing of that history in a repository. We all make mistakes and rebase can be used to correct them. Fossil just takes the other view that we all make mistakes and we must live with them. I guess the choice of which to use very much depends on the circumstance. I use svn, hg and fossil choosing fossil more recently for my personal work for the simplicity of having code and wiki in a blob I can easily move.

On a side note I'd probably refrain from using the phrase "polished the chain" due to the euphemistic nature of 'verb the noun' statements and my natural British tendency to avoid talking about sex explicitly and thus think everyone is talking about sex implicitly.

In Fossil it's not impossible to remove offensive content, though deletion is discouraged. Fossil's policies and content removal procedures are discussed here: http://fossil-scm.org/index.html/doc/tip/www/shunning.wiki

It's also possible to access everything in the repository using the command:

    fossil sqlite3

which opens a command line accepting sql queries. If one knows how to modify the database, then tickets, etc. can be edited. Probably, using the "official" methods to remove content is a lot safer.

FWIW: simply removing records from your repo db (in particular the blob table) _will_ corrupt it. Fossil stores deltas and other metadata which span blobs, and it's not trivial (and in many cases not possible) to separate them post facto. Hypothetically it would be possible to "pop" the top-most change from a repo, then repeat that, going all the way back to the start of the repo, but it's not possible to remove something from in the middle without the equivalent of a 4-way heart bypass surgery.

i've been working with/on Fossil since late 2007, and within the Fossil team we are not aware of _any_ loss of data except that caused by "system events" or severe user error (e.g. once i accidentally included a repo db in a sed filter, corrupting it).

What about https://www.mail-archive.com/fossil-users@lists.fossil-scm.o... ?

[Richard Hipp]: The bug that you hit had been there for ages - it wasn't new. But it was obscure. You were just the first to stumble over it. Sorry.

The bug in this case caused a loss of uncommitted work.

What do we mean by SCM?

It seems to be that "software configuration management" should be about configuring a program, whereas "source code management" should be about managing source code.

Which is Fossil?

The title was changed to something less accurate than what I originally posted.

You are right, makes it seem like this is for storing config files. Source code management is not enough though since this has built in issue tracker and wiki.

It is weird -- I think SCM should mean "source control management" NOT "software configuration management", but the latter is what the website has.

"SCM" can stand for different words, depending who's uttering it, but where fossil stands is this: it fills the same role as git or mercurial. Its a DSCM (which I read as "distributed source control management"). Fill out the acronym as you see fit.

Edit: typo, grammar

Source code management is a subset of software configuration management ("configuration" here refers to the configuration of the build rather than the end program). https://en.wikipedia.org/wiki/Software_configuration_managem...

It's old school enterprise speak. When software-being-built consisted of a number of "Configuration Items" (CIs), such as .doc files, .xls files and of course heaps of .c files. There would be a dedicated "Configuration Manager" who's job it was to make sure that people wouldn't Visual SourceSafe their files over other people's files. He'd also write the batch files for the build server, assuming they had a build server.

They really do mean source control. Just that it doesn't have to be source code. It can be any "configuration item". Just like with git or svn or mercurial or zipping-directories-and-uploading-them-to-a-network-share.

Personally, I like "version control" for that reason. It just removes the subject of the operation from the name entirely.

Yeah that's really weird terminology for a VCS. I thought it was something like Puppet/Chef at first.

Sqlite uses this for their SCM. It makes sense as this is a good test of sqlite itself.

It makes even more sense since it's the same developer.

Indeed, I had forgotten about that :-)

And everything in Fossil is stored in SQLite, not flat files like most other systems. Seems weird at first, until you think about the advantage of storing things in an ACID database.

Can you expand, because I don't know what the advantage is. Other tools can access this information easily, is that you mean?

Transaction support. No work is left midway. A Git repo might get corrupt in case of power failure. (Never experienced that though)

As far as I could tell at the time, I've had git fuck up on windows. I switched to the prod branch, did pull, and it merged the test branch into prod. Then, I deployed it, because I was in a hurry and didn't notice. Fortunately, it wasn't disastrous. That's when I switched back to using Linux as my primary dev environment.

Not if you use a proper filesystem and disk drives which don't lie to the operating system. These may also cause problems for sqlite. After all, sqlite is only as robust as the legs it stands on (filesystem, drivers, kernel, disk).

I'd still trust sqlite to be more reliable than git. For instance, I've previously had git crash and burn with some extremely obscure error message when disk space ran out.

git's error reporting and command line are not exactly friendly, but reliability and friendliness are orthogonal.

So git crashed and burned. Was anything corrupted?

Now for a question. Say, Mr. Evil Gamma Ray flipped one bit on your data somewhere. Git is guaranteed to detected this. SQLite is unlikely to. There's an actual reliability difference here, and it is in Git's favor.

EDIT: Just to point out: fossil does keep SHA1 hashes of everything, so it can find problems just as well as git does - but sqlite itself generally cannot.

No, it generally cannot.

A git repo is always consistent: when putting stuff into the repo, you add complete files to the "loose object" directory, and then modify the ref while holding a lock (and doing a "compare-and-swap" under that lock).

Which means that, as long as the filesystem does not reorder operations such that later operations are visible before earlier ones (and a sane system does not - in that sense, ext3, ext4 and NTFS are all sane), you are either before the ref change (at the old version) or after the ref change (the new version) - any other state is not accessible.

There is nothing magical about SQLite's transaction support. If the operating system reorders files across an fsync(), both sqlite and git will corrupt. If it doesn't, it won't.

Git is transactional and ACID. And unlike SQLite (or even Oracle for that matter), when "git fsck" says "ok", you know that there is no error in the data (caused by bad memory, undetected disk error, etc). Repository integrity is fundamentally inherent to how git works.

EDIT: Just to point out: fossil does keep SHA1 hashes of everything, so it can find problems just as well as git does - but sqlite itself generally cannot.

Thanks for the detailed explanation, I stand corrected.

Well, yes, but mainly that fact that you get the ACID guarantees for free: Atomicity, Consistency, Isolation, Durability: http://en.wikipedia.org/wiki/ACID

Git is ACID too.

People are talking about ACID, which comes with SQLite for certain, but here's the biggest thing: all that regular data, which has specific relationships to each other, is available in a well understood, battle tested high level language. Not Perl, not Python, not Tcl, but SQL. Some people criticize fossil as unsuitable for the task of managing all the subtle relationships and being able to deftly handle lots of data, because it's written in C. They're missing the forest for the trees. SQL does the heavy lifting here.

Speaking of Tcl, the author of Sqlite and Fossil is also on the Tcl core team.

http://en.wikipedia.org/wiki/D._Richard_Hipp

http://www.sqlite.org/tclsqlite.html

Was on the core team.

http://code.activestate.com/lists/tcl-core/7985/

http://wiki.tcl.tk/977

Edit: add list of current/past members

Git and Fossil ultimately deal with fundamentally the same data in the form of atomic artifacts. git chose the way to store the blobs itself on the filesystem so it had to implement a special file format along with special tools to make sure those formats could be read and written correctly. Basically git's interface to the storage are open() and read() system calls.

On the other hand fossil chose to store its artifacts in a SQL database because it makes interaction much much easier (it's just a SELECT; the SQL engine does the rest). Moreover, since fossil has access to a SQL database, it can cache a lot of information to make future extraction of information easier: instead of scanning the whole database on every read (or building a custom index as git does) you can create custom tables for your use case that point to the correct artifacts. Oh and SQLite is just an implementation detail, the real thing that matters is SQL.

So the real advantage of using SQL(ite) is in querying the data to produce all reports you want: list of commits/checkins, list of changes related to a file, list of issues linked to a given file, ...

That said, when you have an extremely simple data model and you have few different access patterns for which SQL wouldn't be of great use anyway, you can have your own file format as git does.

Fossil was initially written for that very purpose - to host sqlite. That is still its primary target. The fact that the sqlite project is so heavily used gives fossil a great deal of exposure, and Richard regularly updates the sqlite.org sites from the fossil trunk to help insure that we "eat our own dog food" and find problems before they hit the outside world via an official (numbered/versioned) release.

A bit of history:

DR Hipp, who created SQLite, created a wonderful bug tracking system and wiki for it, called CVSTrac, with the moto "low ceremony defect tracking" that was tied to CVS. It was wonderful - I had used it in my CVS days. It was written in C, based on SQLite, was fast and just worked.

Trac was inspired by CVSTrac, but is written in Python, is not tied to CVS, is slower but much more capable and much more flexible (although still "low ceremony" compared to e.g. Bugzilla).

Fossil is DR Hipp's version control system, which integrates content version tracking, wiki and issue tracking (low ceremony CVSTrac-alike adapted for fossil)

I haven't had a chance to use it - last time I looked at it, it was missing crypto parts that are essential in some of the projects I work on. But if Fossil gets crypto done right, or if I stop needing it done right, I will definitely give fossil a try.

I'm all-for supporting alternative solutions, particularly in the VCS space. SVN is not great for some tasks, but for some its still superior. Git is very popular but still not ideal for some workflows/project types. Mercurial is oftentimes a more approachable alternative to Git, but has less mindshare.

Personally I think variety should be embraced. Let each project use the VCS that works for it's specific needs/developers.

With this mindset, while I would have no issue supporting and making use of something like Fossil (assuming it's reliable), I would absolutely discourage the use of it's wiki and ticket systems, in favour of independent solutions that are not tied to the repo and repo software itself.

That last statement needs more justification. Tickets, in particular, are fundamentally tied to the code base. There are lots of things you could use the wiki for, like internal API documentation, that are just a step above in-code comments. Why on earth shouldn't they be stored in the same system with the code, especially when that arrangement simplifies the setup needed to start working on the project?

Code base !== One repo. I agree for some projects this all-in-one system might be beneficial, I'm saying personally I would expect teams to use standard tools for tasks. A wiki and ticket system that works the same regardless of the type of repo you use.

Im looking at this from the point of view of providing infrastructure for teams to work on different projects. A project does not necessarily mean a single repo.

Fossil's bug tracker is not just a separate system bundled with Fossil. The bug database is integrated with the VCS for a reason. Edits to the bug database are tracked, branched, and merged the same way source code is.

For example, let's say you create a branch to work on a new bug fix. In that branch, you make the fix to the source code, and modify the bug status to "fixed". When you merge your branch back into trunk, both the code fix and bug status get merged atomically.

It definitely is possible to create a bug tracker that did these things but was still separate from the VCS. I remember seeing some early-stage bug trackers that stored all bug data as files in the repository (instead of in a separate database) and could be made to work with multiple VCS backends easily. However, since VCSes weren't designed for this, it was sort of of clunky to use.

Ideally, there would be some interface between VCS and bug tracker to make sure they're decoupled but still work well together, but nobody has figured out that interface yet. Until someone does, it's much simpler for Fossil to ignore the coupling issue for now and focus instead on making the best product they can.

Marking issues/tickets as closed based on commit messages in a configured VCS isn't exactly unheard of in regular stand-alone issue management systems you know.

Separation of tickets and VCS: that has been a bit of a stickler in the development of libfossil (http://fossil.wanderinghorse.net/r/libfossil). In fossil the ticket system is married to a custom mini-script engine called TH1. In libfossil we want to avoid any specific script binding. That's turning out to be quite difficult, as Fossil's ticket system is built from the bottom up on top of TH1 (which is not a very flexible language).

> I would absolutely discourage the use of it's wiki and ticket systems, in favour of independent solutions that are not tied to the repo and repo software itself.

I'm interested what you think about BugsEverywhere ( http://bugseverywhere.org ), which keeps bug info in a hidden directory in the repo and integrates well with git. I've not used it before, but am considering it for a research project I've inherited which has no infrastructure (just source code in a zip file). I'd like to avoid proprietary hosting (eg. github) or hosting my own web server (eg. gitlab), so I'm thinking that BE would be a good fit, with hosting on gitorious.org

It looks interesting, and the multi-vcs support is quite good, and I guess in theory in a multi-repo project, you could just have a repo called "bugs" or "issues" if you wanted to keep that stuff separate.

The demo of the Web UI gave a 502 so I couldn't access it and I don't have time to try it out locally right now..

it looks like BugsEverywhere is in maintainance mode (if not dead already)

I like using Fossil for my personal projects. I don't always use every feature, but it's all very small and self-contained and I can just copy the executable into the project root without having to worry about the system environment.

Since so many smart people interested in DVCS are looking anyway: anyone knows whats happening to veracity scm? QA has been offline for months.

It was really promising but now seems more or less abandoned.

As far as I know, veracity is no longer developed. The list has been dead for a long time (last post over a year ago), and the last release is 20 months old.

A shame, really, because they were at least trying new things in places.

I can't find it anymore, but Eric Sink (SourceGear) wrote sometime previously on his blog that they've decided to focus more on the mobile market with a product called Zumero (http://zumero.com/). IIRC Zumero came from all their work on Veracity.

FWIW, I had also hoped Veracity would be developed further.

I used Fossil for a while, and I liked it a lot. But after using it with my team, and having to host it internally (something I didn't have the time to do properly being a startup), and programmers complaining about the conflict resolution process, we moved our projects over to git/GitHub. Honestly, I haven't looked back, even though I much preferred Fossil's simplicity.

Can you elaborate on the problems you had with conflict resolution? I used Fossil for a couple of personal projects and made a mental note to try it out instead of Git next time I had to host a code repository on a local server.

"...complaining about the conflict resolution process..."

In Fossil? Absurd. i don't know that i have ever, without help from a git professional, successfully resolved a merge conflict in git. In fossil it's trivial.

I had this idea for years - to integrate wiki and tickets with source control. Probably I will like Fossil.

You might also like Trac ( http://trac.edgewall.org/ ), which I've seen far more projects using than Fossil.

Trac is very different from Fossil though: Fossil is version control software with "embedded" wiki and issue tracker, while Trac is a wiki and issue tracker which can, optionally, interface with one or more external vcs systems. Fossil stores the wiki and issue data in the repository itself (SQLite), while Trac stores it in an external database (PostgreSQL, MySQL etc.. SQLite would probably work too!).

I don't mean an application with tickets, source control and wiki.

I wanted to express (implement) tickets as wiki pages, and store them in source control; instead of 3 separate concepts.

BTW, I don't like how trac tickets work - it's impossible to do agile (statistical) planning with them.

Zed Shaw uses Fossil in his Peepcode Play by Play episode. If you have a subscription to Peepcode (now Pluralsight[1]) you can listen to him talk about it and why he likes it.

[1] http://www.pluralsight.com/

And also why he stopped using it when it hosed his repo:

http://www.mail-archive.com/fossil-users@lists.fossil-scm.or...

It did or he did?

There was a previous bug, and it was fixed then, but Zed decided eventually he can't afford to have it happen again (after losing 3 days of work) so he switched to Git.

Richard Hipp (Fossil's author) was able to reconstruct what happened, more or less, here: http://www.mail-archive.com/fossil-users@lists.fossil-scm.or...

I can understand the frustration of losing a couple of days of work, but wow, that thread does not show Zed Shaw in a good light. This approach of "there was a bug so now I'm throwing all my toys out of the pram" isn't particularly constructive or mature.

>that thread does not show Zed Shaw in a good light

It seems that Zed Shaw does not show Zed Shaw in a good light. That behavior appears to be the only kind he displays. A single bug that did not delete anything and all his work can be recovered with a single undo command and he wants so badly to turn it into drama, while insisting he's the only reason their program exists.

It deleted his uncommitted code.

I would have done exactly the same thing. He is using a SCM that deleted all its files due to a bug. He tried to recover its deleted files, and it seems the next commands utterly killed any hope of recovery. What was him supposed to do? keep in its mind perfectly the internal state of Fossil after an unknown bug to find which command, if any, could recover its files? I would have panicked, tried every command I could think of and started crying.

After Fossil deleted its files he informs the maintainer of the bug, giving a clear description of what happened, and decides to switch to a more mainstream SCM.

I don't get why you're giving him so much heat. He seems to me constructive (the bug was fixed) and mature (he went to a more mainstream SCM that is less likely to have some kind of "never seen before" bug while you just "make a branch").

For next time:

    cd ..
    cp -R wrkdir backup
    cd wrkdir
    fossil dickitup

> What was him supposed to do?

Use some automated backup tool that had nothing to do with fossil.

We are talking about Zed http://programming-motherfucker.com Shaw right?

wow, for sure I won't wear that t-shirt, I'd be afraid to get myself killed by some myopic officer or something like that

Also the "if it wasn't for me you'd still be four guys on a toy project" quote.

Doesn't seem to be connected to reality. I've heard of Fossil and sqlite, but never of Shaw, who apparently wasn't even a commiter.

You have never heard of "Learn Python the Hard Way", the Mongrel server or "Rails is a Ghetto"?

He used to be important in the Ruby community as the author of Mongrel. But yeah, if it's not massively hyperbolic,it's not Zed Shaw.

There's more to a project than being a commiter. Shaw promoted Fossil, and people (a lot of people have heard of Shaw) looked into Fossil and started using it because of him. In that same thread you'll see someone else on the list spoke up as being one of those people.

That quote you gave is a reaction to the dismissive attitude that non-commiters are somehow second-class citizens; which you've just expressed. It's a bad way of thinking about people.

Wow, Richard Hipp is a class act: http://www.mail-archive.com/fossil-users@lists.fossil-scm.or...

> "fossil export | git fast-import" will get your stuff moved over efficiently. Thanks for hanging out with us. I expect you might come back to fossil someday, and if you do you will be welcomed.

Zeds reply is interesting as well:

Yep, that feature is awesome. :-) And yes, I think if it you come to me and say it's all stable now then I'd easily give it a try, but I have to be practical and get shit done.

In other words, it's nothing personal, just business.

A disgruntled Zed Shaw must be how you battle test people's tact and composure.

"...complex sql to remove a spam ticket..."

that was very possible the reason for his woes. Do not EVER manually edit the fossil repo db. That's like hex-editing an MS Word Doc.

Looks crazy compared to checking /etc into git.

Fossil is a general-purpose SCM tool, just like git, just with more features (integrated bug tracker, etc), so in theory you _could_ check /etc/ in to fossil as well.

Fossil is not the right job for that - there's a dozen entries in the ML archives explaining why. Short version: it doesn't do file permissions and system users, and many files in /etc require specific perms and owners.

Thanks for the correction, my knowledge of fossil is very shallow

The first thing I notice is how ugly the website is - if this reflects at all upon how ugly the scm is (and it may well not), the it's not going to gain traction.

Ugliness is subjective: I really liked it, or more precisely, nothing about it annoyed me, and many of the design decisions on the web these years annoy me. You got me curious what exactly about it struck you as ugly. (You could reply to my email, which is in my profile, if you are worried about getting downvoted.)

i suspect he's talking about the general aesthetics of the UI. None of the developers (and i'm allowed to say this, being one of them ;) has demonstrated a particular gift for making apps look really pretty.

Reply on news.ycombinator.com