Beyond CI/CD: GitLab's DevOps Vision (2017) | Hacker News Reader

Beyond CI/CD: GitLab's DevOps Vision (2017)

about.gitlab.com

256 pointsFooBarWidget71 comments

We have explored GitLab in the past and a lot of our CI/CD is heavily inspired from GitLab.

They seem to be consistently taking right decisions at the micro level. Their CI/CD design and execution is way more usable and reliable than <name suppressed> pipelines (still no manual stages, still no re-triggering, etc). Their design and integration with Kubernetes is also a great choice.

So on the micro picture, things are quite good. On the macro picture, they are building a universe. On the Issues front, they're trying to be like Trello (and in some places, reminds of Jira). They're trying to tie Issues with customer support, getting slightly in the way of Zendesk/Freshdesk. They're building deployment. Then monitoring - they support Prometheus. Now post-deployment/post-monitoring. And of course, they are competing on core git hosting as well.

Of course, they'll do a great job. Heck, if they get into the messenger domain, they'll pick a nice strategy and just integrate deeply with Slack maybe.

But the core of the problem is: None of these are instrumentable/hookable. Want to enforce some of your own organization policies before deployment? Want to use everything _except_ with your own monitoring tool? Sorry. If you use GitLab, you use their universe and everything that comes with it. There is no graceful integration with a broader set of tools.

Maybe for a small startup, just "doing things the GitLab way" and subscribing to all their micro choices (which are good) make sense. But organizations grow, and they'll outgrow these micro choices sooner than later. Then the lack of extensibility, hooks, etc. will bubble up.

All this is assuming GitLab continues to build out a perfect application platform with all the right choices (and all the version and time matrices of those choices). Hugely laudable work so far though!

On the topic of messaging, GitLab acquired Gitter about a year ago:

https://about.gitlab.com/2017/03/15/gitter-acquisition/

Edit: I first said "not too long ago" then realized it was just about a year ago.. Where is time going!?

Also, omnibus GitLab includes Mattermost, and there are integrations between both (notifications, slash commands, auto channels...).

Their open source offering is so great that I'll gladly accept that they're trying to hook customers this way.

Like the other guy said: If you don't like them adding exclusive extra features, pay them for what they're already offering.

> None of these are instrumentable/hookable.

I think there goal is to support you when you want to use non-GitLab tools. I guess there's probably always going to be hooks missing, but they do have things like webhooks. Might want to lodge a feature request for missing hooks, as I do think they're open to them.

> None of these are instrumentable/hookable. Want to enforce some of your own organization policies before deployment? Want to use everything _except_ with your own monitoring tool? Sorry. If you use GitLab, you use their universe and everything that comes with it. There is no graceful integration with a broader set of tools.

Huh? Gitlab has a huge list of integrations[1] and webhooks[2]. You can use Jira for issue tracking or Jenkins for CI, for example. If something is missing, it's open source, so the plugin can be added by the community if need be. The software itself is hugely configurable for different workflows and team structures. Can you be a little more specific on what Gitlab is doing poorly here?

[1]: https://docs.gitlab.com/ce/user/project/integrations/project... [2]: https://docs.gitlab.com/ce/user/project/integrations/webhook...

I like gitlab, but a mixed open source / closed source product is always going to be a challenge in terms of hearts and minds. Take this quote from the article.

> The other way to look at it is that this is pretty advanced stuff, and frankly, it doesn’t deserve to be, free, open source.

So is all the stuff that gitlab builds on, like git, or ruby, or linux, that's all pretty advanced stuff I'd say.

Wow that really is a poorly worded phrase. If someone from GL is reading this, please fix that, it makes you look pretty bad.

Just be honest and say you want to charge for advanced features, because there's little money to be made in open source and you're a business after all and want to pay your employees a living wage.

>it makes you look pretty bad.

But they are being honest, they are saying what they are thinking....

Being honest and 'looking petty bad' are not contradictory.

They could rephrase it to: "we consider this an enterprise usage pattern for our software so this feature will be included in the paid Enterprise Edition".

It says all those things your comment does. This is what happens when you dump an entire transcript of an hour long video, someone takes one poorly phrased line.

Yeah, that was poorly phrased, sorry.

One of the awesome aspects of being transparent by default: taking an internal conversation between three coworkers and making the recording and transcript public. I love being able to share our strategy publicly and talk about it openly.

While I can't change what I said, we can make the blog post more accurately reflect our beliefs as reflected on our Stewardship page[0]. There's no limit on what "deserves" to be open source. And, imho, GitLab has open sourced a ton of advanced functionality already.

But "GitLab Inc. is a for profit company that balances the need to improve GitLab Community Edition (CE) with the need to add features to GitLab Enterprise Edition (EE) exclusively in order to generate income."

[0] https://about.gitlab.com/stewardship/

> The other way to look at it is that this is pretty advanced stuff, and frankly, it doesn’t deserve to be, free, open source.

Did they actually say that...? I only went to the page to confirm this and it appears true. Sounds like next time I choose self-hosted Github alternative, I should exclude Gitlab from my options - based on their approach so far, it feels like they really do mean this kind of contemptuous attitude of "only basic stuff should be free software". I used to cheer for Gitlab, but this is just way over the line.

And despite what some say, I really don't think it's that great a product - they seem to be focused on getting a million of things poorly than get their core fixed. Performance is still terrible, see what happens to your browser when you view a diff for a large merge request. Also, you there's a limit to the number of commits in the history you can browse - come on, my `git log` loads the whole history in a split second but you can't render it? And they clearly don't care - I reported many issues to them that got confirmed and most of the time those were just shrugged off. As long as you're not an EE user, you won't really get much more than the annoyance of having odd version numbers make the sidebar sticky and even ones undo that.

In your opinion, what is a valid reason for software to be non-open source? Sure, saying the software doesn't "deserve" to be open source might be a bit extreme, but I keep seeing the sentiment on HN that free and open source is the only allowable route and people who want to charge for their software are to be demonized. I'm curious under what circumstances you think it is allowable for a company to profit off of software their engineers were paid to write?

There's no reason for any software not to be open source, aside from greed.

There seems to be a lot of confusion around free software and open source. You can sell open source software (as a matter of fact, you can sell free software too). First of all, open source only means that source is available - the license is up to the developer though and it might as well be paid for commercial use etc. "Free" has two meanings - "free of charge" and "free as in freedom". I'm not arguing for "free of charge" - while it's nice to have software that you don't have to pay for, I could imagine, say, buying Microsoft Windows if I had access to the source code and could hack it as I go.

"free as in freedom" means that I can not only access the code, but also redistribute it and use it in any way I see fit. This doesn't mean that the developers can't sell it though - they can sell support, extra features, hosting and so on. There are ways to profit from free/open source software. I could as well imagine Gitlab Foundation and Gitlab Ltd, where one provides the core and the other non-free modules that cover enterprise use cases. I don't like their current scheme though.

Look at what the free version offers. They are extremely generous with what they are putting in it, and it's super high quality. I'm very happy with it and I think they are doing everything right. Their platform is a joy to use.

I disagree, but it could be that we're using Gitlab differently. I used to be very enthusiastic at first, but over time I got annoyed by its quirks and decreasing amount of attention to CE. Could you tell more about how you use Gitlab? For example, is it self hosted?

The formulation is a bit weird. I can't find it right now, but I recall reading in the past that features that are typically only useful when an instance has 100 or more users are the ones that are not made open source. That said, they've open sourced often requested features [0].

[0] https://gitlab.com/gitlab-org/gitlab-ce/issues/34591#note_56...

I had a similar reaction, that quote seems extremely ignorant and tone deaf. Open source is all about shattering that mentality from Day 1; making an open source POSIX compatible OS.

Thanks for pointing this out. Changed with https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/....

Advancedness is not a criteria in open sourcing or not open sourcing. There are advanced features that are open source, such as Review Apps[0]. There are basic features that are proprietary, such as File Locking[1]. The criteria we use to decide which version the features go in are documented on our stewardship page[2].

[0] https://about.gitlab.com/features/review-apps/ [1] https://about.gitlab.com/features/file-locking/ [2] https://about.gitlab.com/stewardship/#what-features-are-ee-o...

Is it just me or is gitlab becoming more and more bloatware? Is it realy a good idea to bundle everything in one application? Why do i need an integrated artifact management with solutions like sonatype nexus? Is it worth it to add an "awesome environment for ops" on top of kubernetes? why add this complexity?

Dont get me wrong, i like gitlab and use it since the beginning. I just have a problem with this "munch it all together" style of products.

Gitlab in a way suffers from being open source. Their open source product is so good and includes so many features (even a full featured CI/CD pipeline!), that they need something big to justify their paid version. So I don't blame them. You can disable a lot of stuff that eats away your memory.

Corollary: do you hate Gitlab becoming bloatware and integrated suites? Then pay them! Even if you only use their open source offering, pay them. And not just "pity money" like 100 USD per year, but really pay them as if they're a serious commercial software vendor.

Yeah I know people won't do that. But think about this for a second. It does not apply only to Gitlab but to lots of independent open source software vendors too.

How is that going to solve anything? The more you pay them the more bloated version you're going to get. In fact, if you want to travel lite the community edition is actually your best choice there.

The item it would solve is that if you "pay" then you can more directly influence their road-map. As a customer they want you to be happy and to remain one, that gives you the ability to influence direction. Individually, it may not be a lot, but in aggregate customers have power.

The more you pay them the less they have a reason to make things bloated for the sake of making it sellable to enterpises.

Big corporations also really love bloatware because you can tick off so many feature requirements. The resulting vendor locking compared to a solution comprised of multiple small tools is usually not considered.

Is it that it's not considered, or that getting a whole suite of "work nicely together out of the box" tools from a single vendor is looked on as a better option in most cases?

Especially when things go wrong, having one contact point can be quite valuable, instead of having to go through the cat-herding that is getting someone to agree it's their component that's the problem.

> You can disable a lot of stuff that eats away your memory.

Can you give some examples, and/or a link? I like using Gitlab as an easily portable, one-docker-container-and-you're-running hosting service, so it would be nice to make it more lighteweight.

In the gitlab config file you can disable Mattermost and Prometheus which both use quite a lot of memory.

Maybe more like all-in-one platform instead of bloatware. AWS as a stretch for an idea? Obviously they'll need to have high execution to be able to maintain such a diverse set of features. I rather like it right now for an end-to-end solution.

It's their USP; there are already less full-featured open source solutions like Gogs and Gitea. Once you start adding features like pipelines, it's hard to draw a line between what goes and what doesn't. I guess this vision is where GitLab draws it.

Slightly offtopic from what the article focuses on, but the thing that bothers me about all these built-in CI/CD things in software like Gitlab is that people seem to be perfectly content with building an artifact and then just ... pushing it out.

In traditional deployments that may be some kind of "copy this thing over SSH and make it go!", in Kubernetes-land it's more like "lets just modify this API-state to point at the new image tag!".

Neither of those produce a consistent record of changes applied and state mutation like that is very error-prone.

We actually use Gitlab's CI at work, but we have our own deployment solutions built on it that end up making git commits into the NixOS & Kontemplate repositories which then run their own pipelines to deploy.

This way we can always answer the question "what set of applications at which versions was deployed at $time?" and also roll back consistently.

Sorry but it's not clear what you are saying.

What is the fundamental difference between modifying your k8s deployment to point to a new tag and your solution?

You can easily roll back (except with database migrations) and easily see what version is deployed when.

The idea of keeping all state around your deployments in git is being called GitOps: https://www.weave.works/blog/gitops-operations-by-pull-reque...

Infrastructure as Code is a useful thing and this is essentially building on top of that idea. For example, you can spin up an exact copy of any environment automatically including all software running on it. If something utterly takes out an environment you can spin up a new copy of it and be certain it's identical to what was there before. You can rollback and manage multiple pieces of software at once if there's dependencies between them. You have an automatic and immutable audit log of who did what and when. Using git you can keep track of all infrastructure related changes in one place instead of trying to piece together information from multiple sources when debugging.

When you update a Kubernetes deployment in the API directly you're essentially modifying a global variable. If you do this as the result of some imperative process (e.g. a CI pipeline), you don't have any record telling you what the value should be - you only know what it currently is.

You're also only modifying a single piece of your whole state at any given time, meaning that if you have the services A and B and their deployment pipelines independently modify state - you have nothing that declares any relationship between which versions of these should be deployed together.

There's a little piece of infrastructure wisdom I've learned over the years, I refer to it sometimes as "tazjin's law":

Any infrastructure component not controlled by a reconciliation process will eventually fail.

Versions of dependent components will get out of sync, configuration is being deployed independently of the application, and so on.

In order to reconcile your current state with your desired state you must know what your desired state is.

Does that explain it?

That doesn't explain it. You always need some global variable saying what the current state is. In your case, maybe it's the branch pointer in your git tree. In the k8s deployment, it's a tag value.

No, those don't contain the same information.

Our Kontemplate repository contains the state of an entire cluster, including all versions and all configuration.

A tag value is a single piece of mutable data that is in no relation to the other relevant data.

The tag points to the current version of your docker image, which contains the current configuration, except secrets which you don't want in a repo. Older deployments point to the older docker images which contain the older versions.

Still don't see any advantage to justify the additional complexity in your system.

We have more than one service, more than one environment, and auditability requirements (due to being a financial institution).

I find it more complex to try and keep track of remote state modifications than to have a single source of truth, but whatever floats your boat ;-)

You can export k8s state as JSON/YAML and put that into a git repo if you want.

I had this exact same problem with the way i & my team approached our kubernetes clusters. I come from a network management background and in that space we all know of a tool called RANCID. A simple tool that logged into your switches/routers, made a dump & removed counters/oscillating data and stored it into CVS.

This predates infra-as-code.. it took years before someone wrote a clone that used Git instead of CVS

Made something stupidly similar for kubernetes: https://github.com/pieterlange/kube-backup

What about Helm?

I aim for as few moving components at runtime as possible. Helm has tiller, which I'm not a big fan of.

In addition its design (with warts like the separate versioning layer of charts) adds complexity that I don't think is necessary for most use-cases.

This is awesome; it's something that I've slapped together out of disparate components (that've then gradually fallen apart) so many times I've lost count. Having a tool which can manage it all -- from commit, to builds, to deploys, to monitoring -- is a massive boon for small business (and personal projects).

I look forward to the next releases of Gitlab!

Now I’m pretty sure GitLab has a focus problem. It’s already very complex with somewhat unrelated things. You can’t be everything for everybody. This will even further decrease quality of individual parts of the system. It has been unusably slow for a long time. If they try to do everything, they will have way bigger quality issues than slowness.

I'm upvoting/adding this in my favorites if only for the pipeline diagrams.

Dunno whether their vision will pan out though. I do not use gitlab so I'm probably not that qualified to speak but I wouldn't like using one PaaS for all things CICD. I'd be afraid of locking myself in plus loosing a few degrees of flexibility.

I don't get the beyond part, especially monitoring.... will Gitlab only have a Prometheus view in their UI or do they want to create their own monitoring? And when using own monitoring... this is a bit crazy... we have from day to day more and better Prometheus for that.

We're not going to create our own monitoring; we'll use Prometheus and other open source tools.

We are indeed leveraging open source tools like Prometheus, and not seeking to build our own. We do believe however that the data and insights that Prometheus provides can be much more impactful to an organization if it is provided in the workflow that developers already use, rather than a separate tool/UI.

For example when looking at a CI/CD deploy to an environment, you can easily access important Prometheus metrics and in the future logs, from the same console.

This also allows us to build more intelligence into the platform, as GitLab is more aware of your application and its health. One example is to incorporate Prometheus monitoring to compare the performance of a new release in an incremental deployment, and automatically pausing it if key metrics have degraded.

Is anyone here using the Gitlab CI/CD for anything a bit more complex?

I'm especially interested in a pipeline with parallel executions of different kinds of tests, maybe some manual checkpoint in there, a more complex chain with execution on different kinds of hosts or containers.

Last time I checked (a few weeks ago), Gitlab's CI/CD system is great for most use cases as long as you can describe your pipeline as a list of non-interactive tasks -- some of which may be parallelized -- that each can run in a Docker container.

For one of the software projects I'm involved in, I have more complex needs. We release binaries for multiple platforms so our Jenkins master delegates certain tasks to slaves running on specific OSes. Then at the end the Jenkins master downloads the built artifacts from all slaves and publishes everything to our artifact hosting server. As far as I can tell, Gitlab CI does not support this.

In future CI jobs I may even require user interaction, e.g. I may ask a human to sign off a report. I don't think Gitlab CI can do this.

But if your needs aren't so complex then Gitlab CI is great. It's UX-philosophically similar to Travis and setting it up is super simple, as opposed to Jenkins which is a pain to use.

Can't you put the gitlab runner on whatever you want and set up CI environments to the runner?

The issue is that the configuration format makes complex jobs difficult to define. It's good for the simple case, but versus e.g. a Jenkinsfile[0] where you get a complete groovy DSL that can interact with it's environment across tasks, pipeline, branch, etc with ease it's rough. Having fine-grained control becomes important.

[0] https://jenkins.io/doc/book/pipeline/jenkinsfile/

> a Jenkinsfile[0] where you get a complete groovy DSL

Apache Groovy isn't "complete" as shipped with Jenkins. It's collections API is deliberately crippled so it doesn't work.

You could achieve something like this with tags - https://docs.gitlab.com/ee/ci/yaml/#tags. Though you will need to define each machine individually in the CI file.

> For one of the software projects I'm involved in, I have more complex needs. We release binaries for multiple platforms so our Jenkins master delegates certain tasks to slaves running on specific OSes. Then at the end the Jenkins master downloads the built artifacts from all slaves and publishes everything to our artifact hosting server. As far as I can tell, Gitlab CI does not support this.

GitLab CI/CD can certainly run jobs on different OSes, then have a final job consolidate those artifacts and publish to an artifact server. What part can't GitLab do?

> In future CI jobs I may even require user interaction, e.g. I may ask a human to sign off a report. I don't think Gitlab CI can do this.

GitLab CI/CD has manual jobs so a pipeline can wait for human sign off before proceeding.

The pipelines for Flockademic are somewhat complex, and since it's open source, you can take a look at them: [0]. It's building four interrelated Node projects at the same time, running tests, deploying the back-end projects, then building the front-end project linking it to the back-end one, then running some end-to-end tests and other checks on the deployed code. And it does that for every branch.

No manual checkpoints and different kinds of hosts or containers though.

[0] https://gitlab.com/Flockademic/Flockademic/pipelines

CNCF has a pretty complex CI/CD system called cross-cloud where we're deploying half a dozen CNCF projects on half a dozen different clouds. It's using GitLab and is all open source.

https://cncf.ci/

We're got some pretty complex pipelines, so much so that we've reached quite a few limitations.

You can do all the things you've listed. But you can't build an arbitrarily complex DAG of tasks. Instead, you must order everything into a sequential list of 'stages'. Within a stage you have a list of jobs, which are run in parallel. This might be enough for your needs, but I'd like flexibility to express dependencies for each and every task, Say `t1 -> ((t2,t3),t4 -> t5) -> t6`, etc.

And it's a pain to propagate variables from one task to another, like say setting an environment variable in one task, and getting its value in a downstream task. You end up writing scripts to write them to a file, telling gitlab to make that file an 'artifact', and then reading it downstream.

And then there's the transient errors, all of which are listed on open issues. We're using the hosted gitlab.com, and on some of our CI pipelines, we have a dozen or some stages. The chances of a transient error, .e.g. a runner getting a 503 from gitlab.com, are pretty high.

Can you nest stages?

Say I want something like this:

  1. Sequential: stage "build", build-type agent.

  2. Sequential: stage "tests" (this is just a wrapper/container)

    2.1 Parallel: stage "integration tests #1", integration #1-type agent

    2.2 Parallel: stage "integration tests #2, integration #2-type agent

    2.3 Parallel: stage "deploy and functional tests" (this is just a wrapper/container)

       2.3.1 Sequential: stage "deploy", deploy-type agent

       2.3.2 Sequential: stage "functional tests", functional-type agent

Is it doable?

No, you can define jobs and stages. Stages run one after the others and all jobs inside the same stage will run in parallel, that's it, unless you start triggering remote jobs but then it becomes a lot let maintainable.

I don't have personal experience with this, but it seems to be that OP could have done this using their pipelines and maybe the graphs?

https://docs.gitlab.com/ee/ci/pipelines.html#pipeline-graphs

It is with https://concourse-ci.org

Yes, that should be doable but not quite how you have it sketched out. See here: https://docs.gitlab.com/ee/ci/pipelines.html

I would like to know what the goal for auto devops is...

Will it be able to look at a project and compile it? For example: https://gitlab.com/postgres/postgres/-/jobs

Yes, it attempts to detect the language/framework, and build it. It doesn't work for all languages, and is based on Heroku buildpacks so has similar limitations. If autodetect fails, but some Heroku buildpack would work, you can specify it manually. Or, just include a Dockerfile and it'll build that instead.

From the team who know basically nothing about ops? (unless they've hired actual ops people in the last year)

Reply on news.ycombinator.com