Encountering some turbulence on Bitbucket’s journey to a new platform

I was on a project for which I built a kind of git hosting (as a side-effect of, or to support, other features of the product—we weren't a git-host-as-a-service, exactly) and ran into two things:

1) Locking. It's a pain in the ass. You're probably going to need to take it over or otherwise work around it, some how. I never did[0], but likely should have. At scale and with unreliable HTTP operations and all kinds of crazy stuff triggering writes to repos, you're going to end up with locking problems at some point.

2) Caching. Cache the hell out of metadata. Cache entire repo-wide metadata read operation output. Cache commits. Cache archives. Cache, cache, cache. Cache early, cache often.

[0] in my defense, I built the whole thing solo and there are only so many hours in a day, and that was not the only thing I was working on.

[EDIT] this was, like, 2011 or 2012 or something, so there was a lot less info floating around about how to do this, too.

I'm really sorry if it comes across that way. We definitely don't think you did anything wrong. I also get that when things work a certain way for a long time, it's totally reasonable to expect them to keep working that way.

If I could distill the message re: slower merges down to 2 essential points, they would be this: (1) we underestimated how impactful this would be for some customers and that's on us; (2) some, in fact I think many, users believed they need to wait for the merge to complete and we wanted to clear up that misunderstanding.

From your use case, where you merge a small change and then want to pull, create a new branch, and start working again right away, I understand this directly affects you. I am surprised merges would be super slow for you if you're just merging small changes, though; average merge times are still just a few seconds. Have you opened a support case?

For many other users, I do think the UX changes we're rolling out will make a difference. There are a lot of users who would click merge, maybe 5-10 seconds would pass, and they would assume something must be wrong so they'd refresh the page and then it would look like nothing happened. Today we pushed out an update so that if you refresh the page and the merge is still in progress, you'll actually be able to see that.

FWIW we do have some longer-term work in progress that will make merges (along with basically all file system I/O) a lot faster; but it's a ways off and represents yet another significant architectural project (though much less disruptive than this one!). I didn't mention it in the article because it will take a while.

We know that CI build efficacy drops with duration. People wander off to do other things, and either come back to it later than anticipated (all estimates are off by 2x, including estimating when I will check the build again), or forget entirely. Merge-build is one action, and making merges async increases the perceived build time. Perception of sequential delays is a fundamental UX concept. They are seen as a single, longer delay.

That’s not the only reason merges are synchronous, especially in the Atlassian world.

At scale, we have problems with build triggers from commits. Sometimes you have to fire manually.

At scale, tracking deployment failure is painful. The PR you merge may be in a different module than the deployment plan. So I need to merge a PR, then watch the dominoes fall. But the bigger issue is that Atlassian never finished getting deployments up to feature parity with builds. I have several dashboards that report build health, but deployment health is hard to track. It involves more vigilance and frankly it’s exhausting. Exhausting things get dropped every time people feel tired. To work around this, first you do alerts to team chat, then there are too many alerts so they go to their own channel, then people forget to check the channel, and we have a smaller version of the same problem and no solution.

At scale, I can only control whether MY code has enough tests to detect regressions prior to deployment. Breaking preprod impresses no one, even if automation should have caught it. That means a workflow of merge-build-test prior to moving on to the next task.

I don't view this a bad thing; just a sign of recognition for the teams that did the job. Using "we" may come off as if the management is taking credit for the job. Or at least, that's how I see it.

I worked for Atlassian for 3 years. They are a VERY engineering-focused organisation. Both of the CEOs have engineering backgrounds and the whole company is organised around engineering teams. I think you're looking too deeply into some off-handed phrasing there.

> As we planned for this migration

> Our focus was primarily on read operations

> In contrast, we accepted that there would be increased latency for writes

It didn't sound like they were throwing the engineers under the bus at all to me. They seemed to be highlighting that engineering executed well on the items management wanted them to focus on.

I was honestly worried about Microsoft/Github...but I'm pleasantly surprised out how much better Github is now than it was a few years back even.

yes, this is driving me nuts. I regularly have to merge 3-4 pull requests and it's taking over a minute for each. This used to be a few clicks and now it's a 5-10 minute job making sure everything has merged correctly. Then you go and merge dev into a feature branch and have to wait another minute.

I cannot believe this isn't a priority at Bitbucket and we're being told to get used to it and that the UI will hide it. I've been putting off migrating to Github because we use a lot of integrations, but Bitbucket has suddenly become a pain point where it was okay before.

Yes. And data center is a huge pain to work with in a clustered environment, it almost feels like they want you to not use it and use their cloud offering instead.

I'm obviously biased, but we're not an afterthought. I think the best way to explain things is that Bitbucket was a startup that Atlassian acquired; we had to independently solve many cloud-specific problems while the rest of the company was largely still focused on building server products; and as the company started investing in a platform, they prioritized onboarding Jira and Confluence first which are built on a completely different tech stack than Bitbucket Cloud.

The reality is that this migration is one of the clearest signals I can point to that the company is investing in Bitbucket. Our platform teams have been awesome and given us a ton of support, including features that didn't exist before (without going into too much detail... you can probably imagine that Jira and Confluence don't have nearly the same requirements around file system access that we do).

Yes, Bitbucket Cloud and Bitbucket DC are two different teams and code bases; but we work closely together and are talking a lot about both products' roadmaps and future vision. And we even have engineering teams working together on a shared project and may do more of that in the future.

Bitbucket is an acquisition, and I’ve never been especially impressed by Atlassian’s abilities with systems integration. You can always spot the seams.

https://web.archive.org/web/20160303204710/http://www.itwire...

Atlassian makes me long for Trac, which at first was an epithet, but I’ve come to see it more recently as a distilled product that doesn’t distract from the job at hand. There are a lot of things that get done in the tools because the tools have features, not because the tools are good at it, or because it’s a good way to accomplish a goal.

Mostly what I use are histories, and linking between docs, commits, and builds. If confluence had a way to start a task list in the docs and move it to Jira, that would be one thing, but it can’t even do that. It’s all feature factory work instead of workflow-centric, which is what most of the users actually need.

Thank you!

Yeah, running a datacenter is their core business. The UI around git hosting is not that important, despite what people think. (People overestimate UI in general; like medicine, government and sports, everyone has an opinion on how to make a computer UI done right, though strangely nobody can deliver.)

Why?

Don’t worry, yelling and screaming as a larger self hosted customer doesn’t work either!

Don't forget there is currently no official uptime SLA agreement with Bitbucket Cloud.

https://confluence.atlassian.com/bbkb/what-is-atlassian-s-bi...