Bitbucket Down - Major Outage | Hacker News Reader

Bitbucket Down - Major Outage

status.bitbucket.org

71 pointsJake23262 comments

I've said it before and I'll say it again... It makes zero sense to centralize something that was meant to be decentralized.

We need issues, pull requests, comments, milestones, wiki, etc... all to be decentralized. No reason this stuff cannot be modeled using existing Git objects.

GitHub, GitLab and BitBucket are extremely similar. Almost 100% overlap, you could use the lowest common denominator between the 3 and you'd still have 95% of the features.

1. Businesses need a "halp everything is on fire" button that costs them $1000 a second but will fix EVERY issue, (virtually) guaranteed by SLA. Ultimately that is what these orgs provide; the hosting is kind of almost secondary to that.

2. Actually building real-world implementations of "Issues, pull requests, comments, milestones, wiki, etc..." requires top-tier engineer[ing] time and energy to build and maintain. The current Git-on-steroids-aaS approach provides a very successful way to fund that undeniable cost.

> No reason this stuff cannot be modeled using existing Git objects.

That's the easy part.

The hard part is discovery and keeping stuff like teams and moderation when you decentralize it. Advancements in Blockchain-tech, IPFS and similar will make this viable soonish-ly, but I think we are not quite there yet from a technological standpoint.

How is blockchain related to this?

Didn't you get the memo? If you stamp enough buzzwords on everything, it magically works, regardless of their relevance to the application.

I see you got the memo that being dismissive of everything that contains buzzwords is still popular ;)

Both IPFS and blockchain are definitely applicable to decentralizing applications. E.g. DSound[0], a distributed version of Soundcloud is built with IPFS + a blockchain(not necessarily a fan of STEEM though). There could just as well be something analogous for Github, Bitbucket, etc..

[0]: https://steemit.com/music/@prc/introducing-dsound-a-decentra...

Blockchains allow for a distributed, mutable permissioned system that are required(?) for features like "Who is allowed to push to this named repository?".

You can also leave stuff like that out, but then you will end up with a system where everyone can only push to their own forks leaving us with a very splintered distributed system. To compensate we would need a much better system for distributed content discovery than anything I've seen so far.

You don't need a Blockchain for that. You just store the public keys of the allowed users on the repository itself.

Giving readable names to repositories is a possible use case, although I'd just use DNS.

Yeah, I guess you are right, a blockchain is probably not an absolute requirement for that. I think though, that decentralized maintenance of open source project would be a logical next step, and a blockchain-based system could certainly be useful there. This could help prevent a lot of stale/abandoned projects from dying where the only maintainers of a project leave it to rot and there is no one left to review PRs coming from the community.

How would the blockchain help there?

AFAICT, blockchain is the new hammer that is the solution to all of the world's problems.

Sounds like you might like Fossil. [0]

I don't see git reaching beyond what it does right now, but a few others do, though they all involve a learning curve.

[0] https://www.fossil-scm.org

What software aside from Git Appraise (decentralized code reviews) and Git Dit (decentralized issue tracking - alpha quality) do you know of that could be used to move more of what is kept in GitLab/GitHub/BitBucket databases into Git objects and Git tree in order to make Git decentralized again?

> We need issues, pull requests, comments, milestones, wiki, etc... all to be decentralized. No reason this stuff cannot be modeled using existing Git objects.

https://www.fossil-scm.org/ pretty much does this.

That is pretty interesting and does model those things in the same data store. But the data store isn't git objects, as far as I can tell...

"Fossil stores its objects in a relational (SQLite) database file"

Yeah, it fully replaces the whole version control tool.

I think git-the-tool (i.e. the CLI tool) makes sense to be decentralized as it's at the core. If that tool were built around the idea of a central repository (like SVN), then you cannot escape that centralization (without using what are effectively hacks).

I don't think it is wrong to then choose to have a centralized workflow on top of it. There are tradeoffs for either one, so choose what works best for your team.

For example, I couldn't imagine trying to onboard new client employees to a project without having some sort of centralized single-point-of-truth repository. Getting them to grok GitHub (and friends) is hard enough, sometimes.

You could always use all three. That way, if one goes down, there are still two others ready to take its place.

Now that sysadmin guy who told me: "what happens when bitbucket goes down?" when I asked him to move our repos to the cloud, he is smiling. Sometimes it's best to keep stuff in your own servers, if you have any...

not that your own servers are up 100% all the time. But the reserve in that case you could be smiling at the sysadmin guy while he scrambles for a fix, he is just happy not being at fault and having to fix it ;-)

What about when your server and stuff goes down? Or has a bug or issue that causes downtime or similar issues? Your servers could have better uptime than Bitbucket, but maybe not.

We have Bitbucket on our own servers, and we're moving it back to either Atlassian's hosting or to Github. The reason being that we spend far too much time per month hand holding the server when it goes mental and takes all of the RAM and then decides not to server any pages. And it's not for a lack of resources on the server either, 16Gb RAM and 4 core virtual machine, running Linux. So we have more downtime than either Github and Bitbucket combined.

Any guesses why that happens? It's weird that a git server goes mental just to serve a repo. We had gitlab on my previous job and git server on my preprevious job and never ever had server down problems. This period is like 5 years total.

Oh boy, any git down on a Monday is bad. Imagine being in their office right now.

Will be interesting to hear about what happened/went wrong...

Its holiday today in Ukraine. Yet I am at work... Trying to push some code...

Great. I need to push some stuff real soon now. In the last 12 months Bitbucket had an uncomfortably high number of issues. But whenever I think about moving our company code to GitLab or GitHub I envision going into a world of pain with my eng. team.

Has anyone got some advice for pain-free migration to GitLab or GitHub?

We moved over to GitLab a couple of weeks ago. We're small, so the calculus is a lot different for us. The migration basically consisted of clicking the import button and making sure everybody's remotes were changed over. We switched because we got tired of Atlassian trying to sell us all of their services all the time. The systems that make a big organization run more smoothly are a drag on a small team, and the more we saw of Atlassian's offerings the less we wanted them.

Git isn't centralized so there is no need to "move" the code. You can use Bitbucket, GitLab and GitHub all at the same time -- just add multiple remotes, I do this as a part of my normal setup. You can even pull directly between developer machines without the need for any centralized repository. The tricky part is all the tooling around the code (issues, wikis, CI, etc.), but no matter which provider you use for it, they will have downtime.

GitLab is way better than Bitbucket IMO but also has way too much downtime.

Why not both?

Often there's no preservation of past failure states so people can see (for whatever reason) what the failure looked like. Here's what https://bitbucket.org/chromiumembedded/cef/ (a totally random repository that was in my history) looks right now: http://archive.is/g0I6O

First, my HDMI port blows up and have to work on a small laptop screen.

And now I can't access my repositories..

Not happy.

Downvote? Even more miserable.

This is why it is important to keep backups of backups ^.^

EDIT: I upvoted you twice, HN just loves to downvote anything ;)

When you have to painstakingly earn the right to downvote, you just can't feel bad about using it once in a while.

Hi, I've always been curious what the quota/threshold is. My last account (i336_) got to 1000+ (before I accidentally locked it...) and I never saw the downvote button. Is it like 10,000 or something?

I think I got it around 500~. There could be a manual component to it?

Your user page shows you how much karma you need to hit the next level. In your case, you need 172 to downvote comments. https://news.ycombinator.com/user?id=exikyut

I had no idea! Thanks for the pointer. There are more levels?

I don't know. Maybe at 1,000,000,000 you become CEO of ycombinator.

Really? I don't see that, I just see a karma level.

It's 500 last time I heard, but you don't get a downvote button on every comment.

And there is no downvote button on posts (or if it is, I haven't reached that threshold yet.)

Either they tweak the threshold over time, or there are other factors than karma. I'm sure I gained a downvote button well before 1000. I don't think it's particularly useful -- I only use it when I think a comment isn't worth reading at all. I usually upvote people when I think they're wrong, so we can have an interesting discussion.

I'll up the ante, there is a hurricane in Ireland.

And you started a lousy off-topic thread where you and others are competing to complain about downvoting and the weather.

Yikes. What happened to the HDMI port, out of curiosity?

And I presume macOS? If Linux, X11 can do some interesting hacks with scaling and panning. (`xrandr --output ____ --scale` and `xrandr --output ____ --panning`)

Investigating - Following reports from customers starting at 12:45 UTC, Bitbucket Cloud became unavailable. Our engineering team is currently investigating this issue. We will provide an update as soon as we have further information. Oct 16, 12:56 UTC

" Our engineering team has identified the root cause of the issue and a fix has been applied. We are currently verifying that the incident is fully resolved. Oct 16, 13:48 UTC"

The page shows that git over https is still up, so hopefully it's just a peripheral problem rather than a fundamental one.

If you have one of these services you should probably invest in a backup system. Either a self hosted Gitlab or even just a clone of your repositories on a server or like AWS CodeConnect .

It's back for me.

Wow, that was FAST. It's back for me too!

Spoke too soon, it's down again.

"Minor*"

I can't access bitbucket.org, so...

Edit: / works, repos don't

Sorry, sarcasm alert. They are showing "We ran into a minor glitch.". I definitely don't agree with their assessment.

Ah, thanks for the clarification :) all good then

What does work is

    git clone https://username@bitbucket.org/organization/repo.git

so you can probably still get by. Just the website is down.

Oh! Interesting, this is good!

All the various CI systems and scripts with hard-coded SSH-based git access will be hard to change to use HTTPS for just a few hours.

This doesn't work for me. (Edit: and, it's moot now, because it seems to be coming back up!)

I also have 2fa enabled (it's a private team) and my repo is private... I do seem to reach the repo, and it tells me I'll need an app-specific password to proceed.

Too bad the interface to create one is down.

Reply on news.ycombinator.com