Adding Mercurial support to Gitlab
gitlab.com
gitlab.com
This results in 90 minutes of waiting with hg pegging one CPU core (CPU-bound, not disk-bound!), then a few minutes of hg actually writing files, then the actual build taking 60 minutes.
In the git community, it was always taken as a sign that they were doing something right that `git clone` was faster than `cp -r`. I'm having a hard time not taking it as a sign that Mercurial is doing something wrong that `hg clone` is so much slower.
edit: The saying was that `git checkout` is faster than `cp -r`, not that `git clone` is. The sentiment stands.
When you cp an active git repository you receive an identical file state; when you git clone you receive only what is necessary. Old objects, reflogs, et al aren't in the clone. A clone of an existing local repository may also make hard links, if your file system supports it.
https://apenwarr.ca/log/20080121 (some more nuance)
--local, -l
When the repository to clone from is on a local machine, this flag
bypasses the normal "Git aware" transport mechanism and clones the
repository by making a copy of HEAD and everything under objects and refs
directories. The files under .git/objects/ directory are hardlinked to
save space when possible.
If the repository is specified as a local path (e.g., /path/to/repo), this
is the default, and --local is essentially a no-op. If the repository is
specified as a URL, then this flag is ignored (and we never use the local
optimizations). Specifying --no-local will override the default when
/path/to/repo is given, using the regular Git transport instead.From what I could tell, whatever the slow part is, it wasn't parallelized at all; of 16 cores, 15 were idle, and 1 was pegged at 100% (and very low disk-wait).
I've been meaning to dig in to this more. I'd tried adding --stream, figuring that maybe it was re-compressing the entire repo; but that just lead to a warning about "server doesn't support streams" or something like that. I'll definitely try ygra's suggestion to try --pull, and if that doesn't yield good results, I'll actually dig in to the sources and see what's going on.
In any event, if you're enthusiastic about this kind of thing, we'd love to have more eyes on making hg checkouts consistently fast, even in the face of filesystems undermining those efforts. ;)
I think since then many people have realised that Linus was right and that git is good, but it just wouldn't have gained so much momentum without Linux.
Git's concept of branches as simple names for commits, and the ability to switch branches really easily within one checkout (rather than encouraging the use of multiple working directories) helped hugely. Mercurial grew the ability to do git-style branching later, but git had it from the beginning.
Git has always treated performance as a first-class feature, and that has enabled certain types of workflows that Mercurial makes painful. Git is fast enough that many things don't feel like they take any time at all.
Git integrated first-class support for carefully constructing and reworking a set of patches to be sent by email. Mercurial again eventually grew such support, but early on, it was much more insistent that all branches must be kept around, and by the way are you sure you don't want to push your development branch experiments up to the server and keep them there forever since you haven't merged that head into the mainline?
Even later on, mercurial had more of the functionality people wanted but relegated it to "extensions", while Git shipped it out of the box.
Take a look at https://walac.github.io/mercurial-for-git-lovers/ (from 2015) for just one of many examples.
My understanding is that while Git stores a tree of objects indexed by hash (and then packs them into packs along with an index, which can be repacked arbitrarily later on), Mercurial's primary data format is a log-structured list of revisions as deltas.
The underlying data structures used are different, but considered an internal detail, so could be changed in the future. Today, most data is stored in a "revlog", where data can be either be stored (compressed) directly or stored as a (compressed) delta against an earlier version. Deltas may be chained. The index consists of fixed size records, which gives fast lookups, which are also linked to global revision numbers so that you can eg quickly find the commit based on a specific file change.
Obs: This is based on attempting to re-implement the basics of Mercurial from scratch.
So while it might be obvious for you why Git became most popular, I believe that today, it continues to be that way mostly because of GitHub.
As a heavy Mercurial user, but only for the last couple of years, I'm confused by this one. You can just as easily switch between branches (in the Mercurial sense) and bookmarks (which are Mercurial's name for Git-style branches). How did that used to be different?
In fact one of the nice thing about Mercurial is that you can easily switch to any commit and you don't need to choose between a hard reset and a detached head, because you can always make a commit anywhere (multiple heads for the same branch) and it will never get garbage collected (a foreign concept in Mercurial). Mostly this is good because you simply never need to understand what these concepts mean, which it turns out are not fundamental to DVCSs. But also this makes a neater alternative to stash (although stash is also available): just commit your half-finished changes, mark the revision "secret" (which means it won't be included in any push), then switch to another revision. You can always strip or rebase the revision later, so long as you haven't pushed it (admittedly these actions do require extensions, but they're included with all Mercurial installations).
This might be mainly habit and familiarity with a different tool by now, but to me it's simpler than equivalent Git workflows as there's less ceremony involved to start working and there's no fear around accidentally losing commits because they don't belong to a branch (I kinda hate having to go into the Git reflog to find commits that are temporarily lost because they don't belong to a branch).
I've used Mercurial from the very beginning, the branches are great and bookmarks are fine. I see no reason why you wouldn't use them.
We as a community of developers have learned a lot about the power of DVCS since git and mercurial got started. I'd say even the developers of those two tools have learned a lot since then.
So are we going to have something better than both hg and Git? There hasn't been any new DVCS for a long time.
It's a pity, because I like its technical ideas.
Especially the Windows port seemed rather prone to corruption.
Quite the opposite of my perception.
I started with git. Then I got a job and for a while I worked only in Windows. At the time, I looked into git for Windows, and while everyone said it was stable, everyone also said that it was unofficial - the Git development team had stated clearly they were not interested in maintaining a Windows version.
I didn't want to be in the position to justify an unofficial tool to my employer, so I looked at Mercurial - Windows support has been in there since the first day. So I switched to Mercurial. In almost a decade of using it on Windows, I have never had any problems.
Mercurial is great - I've never come across a use case where git is clearly better.
Basically the problem is that unlike Mercurial branches which are actually branches while git branches are just a pointer to a particular commit. That pointer has no knowledge about what other commits are in the branch and the pointer can be moved anywhere across the tree (even to unrelated commits) with ease. Apart from the reflog, it's basically not possible to ask about the set of commits in the branch. The best you can do is never ever move the branch pointer except forward and make a note of the branch that your branch branched off of so you can do things like `git diff my-branch..parent-branch` (`hg diff -b my-branch` iirc). And this is just one of dozens of git warts.
Mercurial usually had the speed advantage but then git came along and wasn’t written in Python. Speed wise it blew Bazaar and Mercurial out of the water.
It was harder to learn for someone used to svn, perhaps, but the sheer weight of both Linus and Linux made it hard to ignore.
I think GitHub deserve a lot of the credit for git’s omnipresence, though. With Launchpad our aim was to create something bigger than “just” a code sharing site and fairly or unfairly both Launchpad and Bazaar were seen as too close to Canonical. Whereas GitHub picked up the mantle that Sourceforge had dropped and made something way more collaborative than Sourceforge but held onto the idea of being a neutral third party.
RCS -> CVS, CVS -> SVN, SVN -> Darcs, Darcs -> Mercurial, then Mercurial -> Git.
Bazaar I had to use for interactions with Ubuntu, but never in a serious way. Similarly I used a few other things such as Visual Source Safe (shudder), SCCS, and even BitKeeper, but I think they were used quite shallowly.
Benchmarks for this are hard to come by, and are for versions that are considered quite outdated now. But from what little there is, there wasn't that much of a performance difference between git and hg. For example, see this: https://www.draketo.de/proj/hg-vs-git-server/test-results.ht...
As for benchmarks, I don’t have any but there’s plenty of anecdata elsewhere in this thread that echoes the idea that git had significant speed advantages over both bzr and hg.
I think it's more that Launchpad is terribad. No matter how much I used it, I struggled to get to the code so I could read it. It looks like it's been somewhat fixes now with a small "Browse the code" link on the project landing site. It used to be like 4 clicks if you knew what to look for.
And when I used bazaar (on Arch in 2004 trying to contribute to Exaile), it would pin a core and make no progress. I didn't look at it again.
Of course, if you understand the underlying data model, they seem far less arcane, but that's the problem with Git that Mercurial doesn't have - you really need to understand what's happening under the hood to get proficient at it, whereas Mercurial has a far saner interface that hides the internals under a better abstraction.
I just know on my team or 5 much less technical people, the concept of version control and how to do operations in Mercury was near impossible and it fell on one person to manage merging files with master.
Also, this was a distance learning course where the head institution (MIT Media Lab) was keeping a central repo of every location's additions, and the changes made the repo so big that we had hard caps on how much data we could upload.
Later on, it fell on me to introduce git to some colleagues. Just walking them through 'checkout','branch','merge','push','pull' got them on their feet and contributing to a repo. If they hit a wall, I could help detangle the knot, but at least we were in it together.
* hg log by default doesn't tell them anything about the current code they're working on; it's just a dump of everything. To get the history of the current working copy they need to pass -f which they are learning in other contexts means "do something unsafe".
* hg gives commits sequence ids that can locally be used in place of the commit sha, but must never be shared across repositories.
In practice at Mozilla I have seen people run into both these issues, causing much unnecessary confusion.
Even considering the additional overhead, we switched to git after a few weeks since the repository was constantly broken, and the two of us that knew git and SVN were desperately searching for the consistency and tools we had in git. At least we could support the team from there on and make it work for them. Not the ideal solution, but the best available to us.
So I totally understand why git is more popular than hg. Until hg could be considered mature enough for larger teams, git already had a huge following, IMO.
Mercurial was ahead of git for a long time as far as usability and documentations were concerned. I still think it's a better match than git for many uses cases. Mercurial makes the simple stuff simple and the complicated stuff complicated (although still achievable), whereas git is more of a mixed bag in my experience. Git was superior for very large projects with many contributors working on completely different features and sending each other patches by email (e.g. the Linux kernel) but in practice 99% of projects, both FLOSS and corporate, don't have this profile. Git is a formula one, mercurial is more like a comfortable Sedan, so to speak[1].
I think git made it mainly because it could boast hosting the Linux kernel development and all the cool kids were on github.
[1] Which reminds me of this "Git is MacGuyver, Mercurial is James Bond" comparison which is probably widely out of date by now: https://importantshock.wordpress.com/2008/08/07/git-vs-mercu...
1. Network effects of github becoming popular. 2. It's faster. 3. Linux uses it and it was written by Linus (must be good in some people's minds).
Written in Rust and based on a sound theory of patches. If GitLab actually merges Mercurial, I sincerely hope Pijul is next.
That said darcs was a pleasure to use, when it worked, and I have fond memories of it if I ignore the speed/corruption issues.
Slow performance under circumstances you and the app can't predict is a lot harder (an unwelcome relative of our good old friend, the Halting Problem).
$ $ git config -l | grep checkout
alias.co=checkout
$ which co
co: aliased to git checkout
Now if pijul takes inspiration from Fossil and implements things like issue tracking as part of the DCVS then that would be an other story as far as I'm concerned. That would be a pretty radical upgrade from git.
Nailed it. And you could argue in an analogous way about all the cryptocurrencies that try to be "better" than bitcoin/ethereum (offtopic I know).
This is the main question everybody should be asking about potential contenders [0].
For most places, switching would be a huge undertaking. Even if there's an import tool that works flawlessly (big assumption), you now have to potentially re-think all of your secondary tooling [1]. Not just 3rd party tools, either. You have to think about potentially tweaking build pipelines, etc.
Not to mention (re)training people which is very likely orders of magnitude more costly [2].
For any potential tool to overtake git, it will have to provide some pretty extreme benefits. Git originally provided some pretty extreme benefits over SVN and yet there are plenty of places that still can't justify the cost of even making that switch.
[0]: I'm only talking about contenders for large projects and/or business' ecosystems. Use whatever you want on your side projects.
[1]: GitLab adding support for alternatives is a good way to start mitigating this.
[2]: In more ways than just purely financial.
Companies that are at rest tend to stay at rest.
darcs was an absolute pleasure to use, right up until they got stuck on the "merge of doom" bug http://darcs.net/FAQ/Performance#is-the-exponential-merge-pr...
Rust runs on Windows, Linux, Mac OS X, FreeBSD and NetBSD. If you are on one of these platforms and are seeing this then please report an issue, along with the following values:
navigator.platform:
SunOS i86pc
navigator.appVersion:
5.0 (X11)
curl https://sh.rustup.rs -sSf | sh"
curl: (35) error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure
I don't want to have to give up Mercurial just because someone somewhere is trying to be trendy by rewriting parts of it in Rust, because he thinks that's all the rage now.
Actually, after a bit more research it looks like Solaris is a lower tier platform, which means there are no prebuilt binaries that are available, but it should work if you build it yourself.
What you're saying is that if I had to upgrade to a newer version of Mercurial which used software written in Rust, I'd have to bootstrap and port an entire programming language, just to use a newer version of my revision control software?
That's why Facebook chose to just fix the slow parts of hg[1], rather than fix the slow parts of git, to build a VCS which would scale with its codebase.
[1]: https://code.fb.com/core-data/scaling-mercurial-at-facebook/
On the other hand, Google seems to successfully use Git in the same way.
Also parts of Google, like Chrome, Android, and Go, do use git.
I simply do not find this to be true. I've used git for years and tried to use hg to participate in a project and I straight-up just could not understand its branching model and how exactly I was supposed to use that tool. I ended up maintaining my work in a git repository and very carefully doing individual commits and a push in the hg repository only at the very end of a long work period. My daily work remained in git because I could not for the life of me understand how people work with that tool with regards to local branches and squashing. (Which, sorry, I cannot do without.)
Local branches are fairly simple - use the phase command to mark your changesets as secret.
As for squashing... It's more complex, and not so easy, though doable, because Mercurial discourages it. (Though the solutions are on their wiki [0], and various extensions can simplify it for you.)
[0] https://www.mercurial-scm.org/wiki/ConcatenatingChangesets
We use this ALIAS:
histamend = !$HG histedit $(hg log -l1 -r '(draft() or secret()) and branch(.)' --template={rev})
This shows you option to drop/squash re-order/edit messages on all draft or secret commits in the current checkout branch
There isn’t a Git workflow that can’t be done using Mercurial; in fact, the Mercurial team from what I’ve read and listened to via podcasts, has intentionally added features to accommodate Git-style workflows, since developers obviously like and are using them.
Does that cause a huge impedence mismatch in the UI? Or will they converge?
In the same vein, I wounder if it's possible for a single repo/history to be checked out with either tool. You clone with hg, I'll use git. We both merge with patches as a common language.
Disclaimer: only use git
But first, the proposal above, in my view, isn’t core mercurial, which is what most of this discussion is centered around. This is mercurial with evolve + topics, which has differences to what many of you are familiar with.
With evolve + topics, many of the critiques of mercurial go away:
* Light weight branches for short development
* Ability to squash / fold - mutability
* Ability to rebase - mutability
That is why this proposal is important and stands out from bitbuckets offering. Bitbucket (For now) only offers first class evolve support (in beta).
With that said, we use mercurial in production for our main repository. We do use evolve + topics for our development and when commits land in production, those commits become “immutable”. Topics were very much a game changer for us when we started using them about a year ago. We had previously used bookmarks for collaboration and that situation wasn’t really viable.
Like most people reading this here, our development team targets mostly web development. Most of our dev lives in one repo and generally we target a deploy per day, or multiple per day. We aren’t “continuous delivery”, but we are “effectively” “continuous delivery”.
We have a straight forward workflow. We have a Topic per Jira ticket. That Topic gets cut from the head of the branch “default”. Developers work in these "feature branch" through the features lifecycle, sometimes these are big things (multiple week - big changes) and sometimes small things (hours or days worth of work). Developers can rebase their work, if they wish, and they can squash, if they wish. I generally discourage the later as I’m a believer more commits the merrier. I’m also not allergic to merging and I don’t see the value in a purely linear history, so rebasing isn’t as common for us. We also have a single integration branch. When developers are ready, we merge to this branch and the ticket gets advanced to qa. Once the ticket passes qa, we merge to default (therefore advancing the head) and the topic is closed out on its own.
So with the above as a brief overview of our process and workflow, I wanted to also talk about some of the pain points that mercurial has. Some of these aren’t necessarily caused by mercurial directly, and some aren’t even addressable (ie, if we were to switch to git, we’d have the same problems, or similar)
Pain Points:
* Tooling kind of sucks. The biggest argument now to use git is less git itself, and more that you are buying into an ecosystem. As an example, we use Buildkite for our CI. Getting this started required some hacking to make it work. Buildkite doesn’t “just work” with HG. That’s generally the tooling story. We can make some stuff work, but most stuff doesn’t “just work”.
* Merges can occasionally still be painful in a collaborative environment when multiple devs are touching the same files and the same line numbers. I think this is an unsolved problem and probably will remain so until the universe cools to absolute zero.
* Evolve is great and I use it all the time, but some of developers here complain that it isn’t as well documented as other pieces of mercurial.
* In my view, the UI is simpler. Our team is comprised of individuals of varying degrees of technical sophistication. For examples, our designers are not software engineers and don’t really need to understand what a DAG is. But they do need to commit directly to our repository, create topics without asking others, and really only ask for help when they did something wrong / got themselves in trouble
* TortoiseHG. This is a great tool, and most people can get benefit from it. It has some support for topics.
* All commits are kept for all time. Not everyone finds this valuable. I find this extremely valuable. Evolve handles the mutability question by “hiding” old commits and basically creating new ones.
* Hg log, which was mentioned before as a burden, in my view is a huge benefit and feature. Most hg commands take a revset.Personally, I've only used Mercurial to check out some projects that use it exclusively (i.e. (previously) Python and Firefox) and to create a toy repo of no significance (to learn its CLI interface).
Just to highlight a couple things from your comment:
> Therefor, if you value some of mercurials core features, and your primary value from git is the light weight branching model, mercurial with evolve + topics starts to look like a real solution for many shops.
By far the most annoying thing about going from git to mercurial is the overloaded concept of a 'branch'. I couldn't care less about the continued existence of my one-off, topic branch. TBH, I prefer the git concept of the reference pointer being culled in the process. The (git) branch's existence continues on only in the merge commit message. Although this assumes 1) an actual merge was committed (not rebased), 2) the merge wasn't fast-forwarded (--no-ff), and 3) the default merge message format was used.
> With that said, we use mercurial in production for our main repository. We do use evolve + topics for our development and when commits land in production, those commits become “immutable”.
> Developers can rebase their work, if they wish, and they can squash, if they wish. I generally discourage the later as I’m a believer more commits the merrier. I’m also not allergic to merging and I don’t see the value in a purely linear history, so rebasing isn’t as common for us. We also have a single integration branch. When developers are ready, we merge to this branch and the ticket gets advanced to qa. Once the ticket passes qa, we merge to default (therefore advancing the head) and the topic is closed out on its own.
We use Github. IMO, Github made a HUGE mistake when they added the options to Rebase and Squash in-lieu of the only sane option, Merge. I suspect that a lot of users wanted these options because Github (still) refuses to drop the --no-ff from their Merge strategy. I prefer --no-ff, but I do agree that it does make for an ugly log graph; Unlike most people, I couldn't care less about pretty graphs.) Rebase and Squash shouldn't be considered "Merge" strategies. They're for cleanliness, but only prior to sharing changes. Having them in Github has made the process of figuring out what has (and hasn't) been merged just absolutely atrocious. I'll leave it at that, but needless to say my arguments haven't convinced my peers' desire for pretty graphs.
As far as other user comments, I think the most egregious suggestion has been (paraphrasing):
> Mercurial has (had) better documentation.
Which I think is patently false. Git' manpages are (and always have been) fantastic. git-scm.com has been around for as long as I can remember. If you really want to do so, you can get yourself deep into the git's technical documentation which will teach you everything about the file formats and algorithms that are being used. All of this documentation has been around since git maintenance was handed over to Junio Hamano (circa May, 2005); Long before it's popularity explosion.
Are there specific tools for that?
* Huge monorepos and thousands of developers
* Repos with huge binary files
* Repos with few developers
* Personal repos.
* CD repos
* Interconnected repos
I hesitate to criticise it, but I don't think such an environment would be polished. A frozen environment is one which can't be improved, so will naturally stagnate. You couldn't achieve a state of being "extremely polished" because those who cared to do the polishing would be denied the ability to do so because it requires change in order to do so.
The obvious counterpoint to your statement is that if your mentality held sway, we would never have had Git and would still be on SVN.
To put another way; I HATE git, I LOVE Mercurial, but Git won, it's ubiquitous and Mercurial is not that much better.
Having one thing doesn't ensure at all that it will be polished. In fact, it's more likely that it will stagnate.
Hg and Git are both GPLv2.
Even if it's not the best way. If it's for more than one person, for the sake of sanity, pick a single way to do normal tasks. It doesn't have to be static (changes to the process/tooling can be trialed and applied), but it should be single.
I've built developer tools for big teams that improved productivity and increased my and others' happiness for a living. And I'd shred every single commit to those tools in a heartbeat in exchange for an environment where you don't have to head-scratch about what process and tools other people were using to make changes to the same code.
I'm actually super happy about mercurial support. Git is very complex and that complexity is a real barrier to entry and a time suck when a beginner invariably screws up. And if you feel comfortable with your git Fu, please explain to a beginner how to recover lost commits using git reflog or when you would use an octopus merge.
git reset --hard
C is just terrible, it has to be improved, but the "Git vs Mercurial" is not the same topic, because Mercurial is not THAT MUCH better to worth to replace it or make thing more complex.
I've used RCS, CVS, Source Safe, Perforce, bzr, subversion and git professionally as a programmer (and that's not including home grown VCS systems that nobody has heard of). I like git, but it's hardly going to be the ultimate VCS ever in the history of the universe.
I remember using RCS and switching to CVS. You have no idea how happy I was! It was like night and day. When I eventually started using distributed VCS systems I had that same feeling.
We need a variety of systems where people are pushing the boundaries in order to get to the next level. I don't think Mercurial is the next level, but there are some really good ideas in it. Sticking to a single implementation will hold our whole industry back.
To be fair, learning a new VCS takes as much getting used to as changing the colour of your sneakers compared to all the other stuff you have to learn in your career. If you hold on too much to a small little island of technology, you could find that the rest of the world passes you by pretty quickly (Speaking as a fairly old guy, a lot of my former colleagues are in that predicament right now -- it's not fun).
RCS provided revision control CVS provided concurrent checkouts by different users Subversion allowed moves and renames and atomic commits git is distributed.
Each one of those was so much better than the previous one that it replaced the previous one, despite that previous one being ubiquitous.
Mercurial might be 5% or 20% better than git, but it doesn't matter. It's not 300% better.
Mercurial won't take over after git, in the same way KDE won't take over after Windows. The next thing has to be the iPhone - the thing that changes the paradigm completely.
But it is better; just how much depends on the situation. If you were the guy who blew away a bunch of Jenkins repos using a “feature” of Git, you might feel differently--https://news.ycombinator.com/item?id=6713742
But even though I'm using Git for everything, we need Mercurial. And any future new VCS system with new ideas. Otherwise we won't know what's possible, where leaps can be achieved. Git didn't come out of nowhere, Linus put together the first version in just a week - he put down the fundament, the basic ideas that Git is built on. The reason he could do that was because he had used Bitkeeper previously (before Bitkeeper he refused to use any VCS at all), and his experience with Bitkeeper enabled Linus to come up with the Git fundamental principles (note that Git is different from Bitkeeper in important ways - particularly the core, where Bitkeeper built on (basically) SCCS and Git has a totally different way of tracking changes).
Git (and Mercurial etc) are what they are precisly because there have been other version control systems before them.
Edit: if you look in the link it seems he has made a comment on the issue:
"This is really interesting. We need to make sure that by integrating Mercurial we do not slow down the future progress of GitLab. Now that we have a clean abstraction with Gitaly 1.0 coming in 11.3 this is a good time to look into it"
Then keep using Git and let others that want to use Mercurial do so. To be honest, it's not clear what your argument is. You use Git so everyone else should use Git?
Almost everyone is using Git nowadays, so the rest should use Git also, yes, instead of wasting brain cycles, even worse huge efforts to support Mercurial instead of improving existing features in GitLab.
I'm not at all against progress, innovation or opinions, but I hate complexity, wasted effort and somebody forcing me to learn yet another tool just because...
Did it have both out the gate and meet criticism?
Dealbreaker for us - we'd rather retrain staff on hg->git (and lose the amazing TortoiseHg).