Repos are disappearing from Gitlab
gitlab.com
gitlab.com
The problem seems to be resolved now, seems like a problem with invalidated cache on a heavily loaded server ( official tweet: https://twitter.com/gitlabstatus/status/897375110205722624) -- not data destruction.
Gitlab Issue: https://gitlab.com/gitlab-com/support-forum/issues/2320
It's unfortunate the expectations we place on cloud providers but my clients have similar expectations from me so I need to work with providers that can assure me they can offer the kind of uptime I'm expected to provide. Plus the modern way of pipelining code means git is seldom a static silo like it once was which only compounds things when one of your links in the chain has multiple glitches a year.
edit:
I thought I would get downvoted into oblivion for this but I felt it was a discussion worth having since they relate directly to my experiences with working at scale. So I "took one for the team" - so to speak.
Anyhow to answer a few points:
* Yes I'm aware git is a distributed VCS but like I said, git is seldom a silo so you'd need to reconfigure your pipeline to point to the new git origin or have your pipeline already configured to accept fallback systems. Quickly your configuration spirals in complexity.
* Yes I'm also aware you can self host. This is actually what we did do before moving to a cloud hosted solution. This added complications with our AWS pipelining. Problems that were solvable but again, with additional complexity compared with cloud hosted solutions.
At the end of the day all problems can be addressed with enough time and resources however the reason people typically opt for hosted solutions is to reduce complexity not increase it.
The reason people opt for cloud solutions is to decrease complexity, not increase it. So if I'm having to run multiple git origins then I would be better off self hosting and just moving those complexities in house.
You're oversimplifying things there. It's not just about the physical instance - I could just run git "naked" without Gitlab's tooling if it were just an issue of installing Git. The complexities arise when you then need to have Git exposed to the internet to allow AWS pipelining to operate. We then have gateway into your internal network that needs to be PCI-compliant (for our retail sites), audited by the Gambling Commission (for our online games) and ISO nnnn (I forget what standard we meet off hand).
It's fair that most organisations might not have these hoops to jump through, but we do and hence why we migrated to hosted providers to remove some of our complexity.
> * The thing to do when you don't want dependencies is to reduce your dependencies, not complain about a third party service that offers their software for free so you can run it yourself.*
I think this is grossly unfair. For starters I'm not complaining about Gitlab, I was opening a discussion about them. You might disagree with my points and I welcome your opinion - otherwise it would be a dull and pointless discussion. :)
Secondly your point about "The thing to do when you don't want dependencies is to reduce your dependencies" is entirely the point I'm making. Perhaps you've missed some of the issues I've raised or perhaps I've explained myself badly. But I have explored a number of options and not just blindly talking out of my arse (which I appreciate there's no way for you to know this given the anonymous nature of the internet).
Lastly why would I be only interested in free solutions given the expectations I have? Currently we do pay for our Git hosting because of our requirements - and my expectations include paying a fee for a service if it solves a few problems. After all we are talking about enterprise systems - any reasonable costs are consolidated and billed back to the client.
No, I think you are grossly unfair. You write that you 'really, really want to like gitlab', harp on them having 'issues every few months' (as if there is any online service that does not have issues every few months, and which blows this particular issue up to a magnitude it does not deserve) and then present a use-case that gitlab offers the perfect solution for, which you reject for reasons all your own.
Sounds to me like you are making a problem rather than using tools in the ways they are intended. Whether the solution is paid or not is not the relevant bit, the relevant bit is that you have the option to self-host which you do not with gitlab's main competitors.
Your AWS complexities are not gitlab's problem, they are yours.
I apologize if I've badly explained the problems we encountered. But please don't assume I'm some idiot who doesn't know what I'm doing just because your business has it's own different set of requirements than ours.
> "Your AWS complexities are not gitlab's problem, they are yours.*
I'm not blaming Gitlab. I'm saying AWS complexities make some solutions infeasible. I also said some of the compliance regulations we adhere to also add complexities that make some solutions less practical. I really don't understand why are you being so abrasive about this.
I said they have 2 issues arise a year (which, by the way, was already counter argued before you joined the thread). I did also compliment 2 other aspects of their business. So to say my "initial comment pans gitlab quite extensively" and "your scorn" is needlessly sensationalising things.
We are all professionals so can we please have a grounded discussion?
I tend to open a bunch of tabs and go through them one-by-one so apologies for the duplicates.
> I did also compliment 2 other aspects of their business.
The proper description of that is 'damning with faint praise', of which your comment is an excellent example.
> So to say my "initial comment pans gitlab quite extensively" and "your scorn" is needlessly sensationalising things.
No, I think it is a quite accurate description of your comment. Whether you agree with that or not is up to you, but just take the fact that I read it as such as a datapoint.
> We are all professionals so can we please have a grounded discussion?
That's an assumption that I won't comment on.
My needs are pretty basic though, no CI/CD.
At some point I've even had multiple remotes in a repo, both on gitlab and github.
If only I could make the same counterargument when my clients complain about downtime. :)
Sadly in the modern world people expect ridiculous uptimes and thus my expectations are just derived from my clients expectations. I don't like that much either but frankly clients often just get their expectations from their customers demands. These days people expect 24/7 uptime so my job is to facilitate that wherever I can.
If your service is taken down by code hosting going away there is issues - unless you are running a code hosting service, or managed DevOps - in which case, the onus is on you to develop for failure (of any provider)
If you can't make that argument, then you indeed have a problem. Downtimes happen, reducing them gets expensive fast. 95% is fairly simple, 99.999 is extremely hard - it requires redundancy on each and every level. Every time a client demands this, I reframe their request in terms of required engineering effort. Few, if any, actually care about the 9 behind the decimal point after that.
The best clients had downtimes already integrated into their operating plans: What happens if we go down, partially or fully, what do we do, how do we communicate and - most importantly, how do we get back up as fast as possible again.
Without wanting to give away too much identifiable information, my clients include multiple major national publishers, a few international sports bodies, and one household console gaming brand.
They are businesses an order of magnitude larger than us and to compete for business you have to make some pretty serious pitches. Sadly uptime is usually one of their primary expectations. In fact one of our sports clients even demanded we have provisions in retain complete uptime during DDoS attacks! (though we did manage to negotiate a compromise on that requirement - thankfully)
With regards to your first point. Yes you're right about how it's probably not in Gitlabs interest to pay for too many 9s. Maybe Gitlab initially appeared to have worse availability than their competitors because they're more open with their literature so I read more about Gitlab's problems than their competitors. But reading a bit more on Gitlab I found a document about provisioning Gitlab in HA environments, which addresses some of the points I'd been discussing.
Which is all fine and dandy - then use a service that provides the required SLA and pay the price (github business or enterprise, have someone host gitlab for you with the required SLA, ...) or store your data redundantly in gitlab and github at the same time (git can do that) and factor the overhead into the bill. I'm not saying a good uptime is impossible - it's just something that will cost (a lot of) money. Bill accordingly.
All the while keep in mind that the SLA of the source code hosting should not have any impact on the uptime of the service itself. Clients - especially larger ones - as a matter of fact do understand that the SLA on issue tracking, code hosting etc. can differ from the SLA on the app.
The issue isn't so much with code hosting but more with code pipelining. We can live without being able to do `git push` for 24 hours(see footnote), but having our code pipeline down during a tight release cycle could be detrimental.
This obviously isn't Gitlab's problem, just a side effect of the current devops automation culture. But something I need to plan for - or at least have considered and weighed up the cost vs impact to business.
Footnote: I'm not implying Gitlab has outages lasting this long. I'm just plucking a figure out of thin air for the sake of a wider discussion regarding HA.
Also a good option - reframe their request in terms of cost. They'll back down a 9 or two when you start adding commas to their monthly invoice.
Gitlab for Git and Rhodecode for Mercurial are so much better.
> Since I prefer Mercurial over git I don't have any other option than to run my own instance.
You do have another option – you can use Mercurial with Bitbucket. Sure, it's not self-hosted, but you were complaining that you were forced to self-host because you prefer Mercurial, right?
Remove downvoting, or increase the cost of downvoting (as in, one downvote will cost you 5 karma points, or something like that) because the HN I grew to enjoy is quickly starting to look like just another subreddit.
edit: typos
Just earlier I got messages that 'info for the latest commits couldn't be loaded' on GH, whatever that means, but I don't see a blog post as to why on HN.
I guess it just goes to show that openness is a double edged sword. Though I'm still thankful that businesses like Gitlab do adopt such policies.
Are you already logged in when that happens to you? If so, you should open an issue in https://gitlab.com/gitlab-com/support-forum/issues with as much info as possible so we can look into it.
If you're getting that when you're trying to log-in, then you're having trouble accessing your GitLab.com account, with which our support team can help you - https://support.gitlab.com