Scaling Bitbucket’s Database | Hacker News Reader

Scaling Bitbucket’s Database

bitbucket.org

86 pointsWalterSobchak56 comments

"Bitbucket uses PostgreSQL with one primary read-write database and N read-only replicas."

I am architecting a project that is at a smaller scale but bears some resemblance, and I’ve already settled that I will be very careful about GET vs POST/PUT semantics and route all traffic of the former only to the replicas and all of the latter through a connection pool to the primary. That, I expect, will hold me well into >1% of their traffic.

I’m not sure if I’ve missed something with their LSN work or if this indicates that the GET semantics horse has already escaped their proverbial barn. Within the call-response of a single request, which they seem to be talking about, none of this should be necessary. Right?

However a cascade of narrowly-spaced follow-up requests could easily catch you in this trick. Lie. Whatever you wish to call it.

Bitbucket is a perfect case for sharding. You could have one DB of users, and N DBs of user data, with the first referencing the latter, and then distribute user data among the latter. You could do this straight up with PostgreSQL without any special server-side software. You could also have a PG server using FDW to act as a proxy for all the DBs a client needs to be talking to. There are many other options too.

Bitbucket (mostly via Atalassian) worked very hard to chase everyone off their platform in the vain hopes that the OSS community would start using Jira. Clearly they bet wrong, but unfortunately they aren't going to get any users back for a very long time.

I've already moved 100 repos from BB over to Github. Free private repos on Github were not enough to get me to go to the trouble of moving, but then BB deprecated Mercurial, so there's no longer any reason to stay. I'm gone.

I just went through this as well. And I discovered that GitHub can import private bitbucket mercurial projects. That made it super easy.

Agreed. Turned out to be much easier than I expected.

I also don't see anything groundbreaking about Bitbucket as a Git repo interface. GitHub is miles ahead in both polish and mindshare.

Free private repos used to be the one thing keeping me (and from conversations, several people I know) on Bitbucket and once GitHub offered those as well, there was no incentive to stay. The Mercurial support was probably another unique feature they had from way back but they seem to have decided to remove that too.

yes, at the beginning, we thought github and bitbucket would be in a healthy competition for features and polish. But then Atlassian bought Bitbucket, and their agenda was, "make BB as awful as possible so we can steal whoever into our crappy corporate software" and that was that. I had to move not just because the issue tracker was garbage but also because Bitbucket's UX became horrible, not just the slowness but also the layout was very broken, having to read diffs inside of a tiny scrolling pane and things like that.

Yep, it was private repos for me. Once Gitlab came about and offered free private repos I bailed as quickly as I could from Bitbucket.

Free private repos is a real upside.

Central login where you can never understand what login you are using and what information they are collecting, nor on which website you are not whether you’ve suddenly integrated Confluence into a Bitbucket instance, is a real downside. God I hate SSO’ed companies, they went all the way of Google into Youtube.

> Free private repos is a real upside.

which isn't true anymore, as other pointed out.

Also offering Mercurial was a reason for choosing Bitbucket, but they are ending its support.

Free private repos are an upside over ... Github, which gives free accounts unlimited private repos? Gitlab, which does the same?

To be fair, Github only added that recently.

github offers these now for individual accounts.

this reminds me that I used to use bitbucket for the private repo and I just hosted some app code on .. github private.

Do you have a link or two for that?

nope just some email exchanges with employees which started with begging them for the most rudimental Bitbucket issue tracker features and fixes and ended with semi-explicit admissions that their hands were tied due to the "just move to Jira" corporate directive.

See "Want a more powerful issue tracker?" at the bottom of https://confluence.atlassian.com/bitbucket/issue-trackers-22... for the documentation of this policy.

Does storing the LSNs in Redis and Elasticache create the same race condition they are trying to avoid or am I missing something?

Yea I'm not taking advice from a service that's down once a week

Did you actually attempt to read the article or did you just come here to spew out needless negativity?

Have you ever actually used Bitbucket in production? It has absurd latencies for every operation from website to even a git push taking full seconds to respond.

It’s hard to take them seriously when that’s the tableau upon which they are advising us to build systems.

A number of years ago I had all of my repos on bitbucket, but recently I’ve pulled everything as their peering can impact large git pulls and were unreliable. The transition with the account changeover was enough to make me jump ship to github completely.

This.

I don't think it's needless. I think the fact that Bitbucket is profoundly second-rate in is quite relevant when discussing their fluffy blog posts about engineering.

I’m not a big fan of Atlassian in general but the blog post provides a reasonable solution to an engineering problem. It doesn’t seem like a fluff piece.

Is it just me or does looking at write logs to determine read consistency just seem like a really bad way to do this vs a different load balancing strategy?

Definitely feels like bandaid, over a more robust solution like sharding their main database by repository. Or splitting off some of their tables to another database. Their load of "millions of requests per hour", doesn't sound particularly high either.

At some point their primary is going to get overloaded again by the writes. And they'll have added all of this machinery for nothing. They've also made the replicas an essential part of their stack. Whereas their previous stack could probably tolerate a certain amount of replicas going down or replica lag. They now have a system that will grow and be dependent on all the replicas being available. Until a heavy write load or heavy table alter, causes the replicas to become lagged at which point a higher percentage of traffic will go to the master and potentially cause downtime.

Yikes. Excellent point. Is sharding the only option here? It seems like having immensely shared Postgres is one way of solving the problem, but is that the only option?

Wondering if eg cockroachDB would be a good fit

This is a very common solution. The biggest advantage is that it allows you to use read replicas which are there for HA and otherwise idle.

It's amazing that Atlassian still exists.

Curious that they use LSN tracking vs making the replication synchronous. I would be curious to see the performance numbers and the reasoning behind the choice.

Synchronous replication really slows down write performance and adds in the complexity of needing to determine if a replica is down. I think very few people use it in practice.

+1, sync replication also reduces availability: any replica downtime becomes write unavailability until you decide that replica is "dead" and stop replicating to it.

There's also hybrid setups of sync+async for durability reasons, sync to a single replica so that if your primary db suddenly catches fire you have a replica that's as up to date as possible, then read replicas using async replication.

Oracle offers DML redirection for active dataguard setup where a write operation on the replica database is redirected to the primary database to allow applications that make infrequent writes to actively run on the Active Data Guard replica database. Also the write operation completes when the replica has seen the write from the primary thereby eliminating the race condition avoiding such complex reengineering

A lot of databases have a concept of consistent reads. Even DynamoDB will allow you to specify consistent reads explicitly on a query and writing through DAX will take care of the scenario as you described as the read cache is immediately updated. This a common scenario.

That's really neat! I wonder, could you store the PostgreSQL log sequence numbers in a cookie to save the round-trip to Redis?

I thought this too, but that then creates a likely unreasonable burden on the client. Not all user agents expect cookies either, think of all the varieties REST clients. Sounds like they would also need to internally reference the lsn too, so being able to look it up internally would be valuable.

Feels like there is something there tho in terms of approach, esp since they are keying it on user id.

I found it interesting that their cloud version of bitbucket uses Django while their self-hosted stack is almost entirely Java.

That's because they bought Bitbucket. The self hosted version was internally developed and used to be called Stash, before they rebranded.

Step 1: Remove Mercurial support.

> we find a replica that is as up to date as the user's saved LSN

The article doesn't explain how they know the current LSN of each replica?

IIUC they need to query replicas until they find one that is up to date.

"We have to query Elasticache and replicas, which adds latency."

10ms to do a redis get? What am I missing here...

They said that was their time budget, but didn't follow up with how expensive it ended up being in practice -- but agreed.

The article says they were able to keep it close to their max budget, which actually suggests that it was higher than 10ms in practice

And presumably query the replicas until an up-to-date one is found (based off the LSN of the user).

I’d have expected the LSN of the replicas to be in redis as well, but fair point. However even as a worst case given their number of replicas, I’d expect to be able to hit redis AND each replica in under 10ms

Still salty about them dropping mercurial support without providing any migration tools, and planning to just delete data without any archiving.

Also salty!

For hg->git migration, https://github.com/frej/fast-export worked smoothly for me.

    mkdir new-git-repo
    cd new-git-repo
    git init
    fast-export/hg-fast-export.sh -r ../path-to-old-hg-repo
    (probably want to translate .hgignore to .gitignore and commit it)
    git remote add origin git@path_to:git_server
    git push -u origin master
    git push --all

Yes, it is fairly easy to convert the repo, but

1) this afaik couldn't be done inside the BB project, so one would have had to create a new repo and lose bug history etc.

2) It's a trust thing. If BB kills repos once, who doesn't say they won't kill others tomorrow?

3) Comfort: Import to GitHub is a click away

Same here. I don't love that they aren't going to support Mercurial anymore. But deleting all our existing projects without providing a path for us to migrate or archive? I can't trust them anymore.

I'm also salty. What's your plan? I'm either gonna migrate to git (and use a different host), or self-host something like hetapod.net

I went to sourcehut. I am very happy, but during the process I ended up actually having to learn a bit of git. Future projects will probably be git.

Yeah I'm moving to git and github. I always pteferred mercurial for what I maintain are good reasons not just of personal taste, but I've been swimming against the tide for too long and it makes sense to go with git and github.

If github goes downhill, at least I'll be in good company next time!

GitHub’s import feature worked well for me. You’re obviously going to lose the wiki, issues etc. But at least you’ll keep your repo history, branches, ...

Reply on news.ycombinator.com