OpenDev is a collaboratory for open source software development at scale | Hacker News Reader

OpenDev is a collaboratory for open source software development at scale

opendev.org

112 pointsxo5vik17 comments

So time for a history lesson.

First, who am I? I'm JayF -- I've been working on OpenStack for about a decade, and am the currently serving TC Chair. This is not an official history, but my attempt to give you the story behind this from my perspective.

OpenDev collaboratory originally started as part of OpenStack itself -- as the team that enabled us to test the giant software suite. When it became clear that the tooling created and used by that team had general interest, it became time for them to spin off into their own group. I'll note that this maps similarly to what happened with Zuul -- a tool created around OpenStack, that was generally useful and split into it's own project. OpenDev was infrastructure services created around OpenStack and for OpenStack, that are generally useful and split into it's own project.

This context is important as to where it puts things on the timeline: OpenDev, and the associated technologies, were not created as an alternative to some incumbant power like Github or Gitlab -- when it was created, there were few, if any, other tools that could operate at that level of scale. (It's unclear to me if Github, as we think of it today, even existed or was at any level of popularity at this point in time.)

So as this thread hopefully picks up steam, lets avoid terms like "alternative to" or things that miss the context. OpenDev has been around, and open, for a long time. It will continue to be around, and open, for a long time. They aren't going to feed your code into an AI to train it, and they aren't going to try and turn your collaboration toolset into a social media platform. It's a tool to get work done, and I've been lucky to work in it for the last ten years.

> It's unclear to me if Github, as we think of it today, even existed or was at any level of popularity at this point in time.

In 2010, when Openstack has had its first release, Github already hosted 1 million repos. https://github.blog/2010-07-25-one-million-repositories/

For comparison: Sourceforge had less than 250,000 in 2010 https://en.wikipedia.org/wiki/SourceForge#/media/File:SF.net....

OpenDev may not have been created as an alternative, yet it may well be an alternative. i get what you are saying, and i appreciate the history. i always love it when people who are/were deeply involved share their insights. but i don't see a problem with calling it an alternative when it actually is one. there is no rule that alternatives need to be newer or be created with the intention to be an alternative.

walking is an alternative to taking a car for short distances, even though walking as a mode of transportation clearly came first.

The UI is much faster than Github, which is great, big selling point for me. The "iterative change" approach is pretty much how companies use Github I think. Contributors are given push permissions for the repo, the main branch being off limits, and PRs are required an approval before they can be merged. The Github model with forks is not great imo, because forks get out of sync pretty immediately.

Some features I would like to see is ranking of repos by popularity, e.g. "Github stars".

> The difference is subtle, but significant. In the pull request model, you create a fork of the original repository, push changes to it, and ultimately propose to merge changes back. In the change proposal model, you prepare a change, and propose it. You do not create a complete fork and everyone contributes into the same original repository with what are basically lightweight, ephemeral topic branches

This is an odd distinction to make. Forking in GitHub is an entirely optional process - in my experience it's primarily used to invite outside collaborators to open source projects. I've never worked on a team that uses this process for their owned repos. I've only ever used GitHub Pull Requests according to the process described here as the "Gerrit way".

I've never used Gerrit so not sure if I'm missing something here: it sounds like they're just describing a process that works identically to PRs with the exception that the platform doesn't support forks?

Quite the contrary to their claim, we restrict forking on ALL our repos and still have meaningful PRs.

Your assertion is correct that most organizations operate this way.

However, open source is different. Often you’ll have someone fork - hack on it - devise a novel change - and request a PR from their fork:branch to origin:branch.

In this scenario, the fork - and subsequent hacking - are exploratory while submitting the change and PR is distinctive. Not sure how one is supposed to explore and hack and provide PRs in their model as you’re forced to branch. A local copy is still a “fork” but with a separate origin.

You can still do a dual origin setup or do a merge main onto yours but I find this extremely limiting to contributors and overly controlling by project owners. There’s nothing stopping you from enforcing squash on merge or having standards on your SDLC. Enforcing that on everybody else isn’t in our best interest though. A diarrhea of commits is how you end up with spaghetti code.

From where I sit, the primary distinction seems to be where the fork lives. On Github, forks are public. You get a copy of the repo in your own namespace on Github which you can pull from and push to from wherever. Others can see it and any changes you make to it as well.

On OpenDev, your clone/fork/personal branch is only on your dev machine and you only push patches, if I'm reading it right.

I guess the OpenDev model discourages people from forking the repo, modifying it to suit their needs, and then not sharing the results. To some degree.

I think I actually prefer the GitHub model where people's branches/changes are public by default unless they choose to go through the trouble to make their fork private. I have found many good bug fixes and enhancements for repos abandoned by their upstream authors this way.

The key philosophical difference is that in Gerrit you are proposing a specific change, where in GitHub you are proposing a branch be merged.

It's a very subtle distinction, but having worked on OpenStack in the past, I much prefer the Gerrit model. Patch revisions are much easier to track over time than pushes to a PR branch during review.

Some of the folks running this (of which I am one) are going to be talking about it in a livestream in a little over a week: https://openinfra.dev/live/

This seems to be squarely aimed at Gerrit fans, which is difficult for me to wrap my head around. The projects I'm on which are Gerrit-encumbered all seem to regard it as a necessary evil rather than a selling point. Still, the project is well-thought-out and I hope it's successful!

From what I've seen, it's more like "what patterns you should follow if you want to do a lot of collaboration with openstack upstream."

OpenStack uses opendev, so anyone who wants to contribute upstream to openstack will need to know about the opendev ecosystem, possibly by dogfooding tools like Gerrit and Zuul in their own infrastructure.

I don't think openstack will go away any time soon, and I don't think they'll stop using Gerrit etc any time soon, so everyone else in their vicinity also remains encumbered with tooling that sometimes feels legacy or over-complicated :)

Personally, I don't mind gerrit as a code submitter and code reviewer. The workflow feels pretty straightforward to me. Some aspects of Gerrit administration are annoying though. It also seems a bit encumbered by being written in java; this flows on to annoyances like not supporting FIDO2 keys yet, and probably not in the near or medium future either (correct me if I'm wrong).

There's an interesting thing about complicated tools: they often exist to serve complicated requirements.

OpenStack is a large, complex piece of infrastructure software used at scale that many people, even on HN, will never experience directly. Why don't you hear about us much? Because people running at that scale frequently don't enjoy talking about the details of their infrastructure.

Ask the question next time you see something in a software tool, e.g. gerrit, that seems over-complicated: What can this software do because of this complication that I'm missing?

I don't think the tooling we use is what everyone should use, by any stretch, but don't assume things are being done in an antiquated way just because they're different than you're previous experiences or because they've been done that way for a long time.

Of course, I wasn't claiming that openstack is doing anything wrong. As someone who uses openstack every day and has contributed (a little) code upstream, it is all straightforward enough for me. One of the reasons we use gerrit in-house is that we like to maintain close ties with upstream, and a couple of our tech leads are core upstream developers/maintainers. I only don't want to name them because I prefer to keep my hackernews identity fairly anonymous :)

My intention was to reply to the commenter who assumed that opendev only exists for gerrit fans, and it seemed like they were missing out on a bigger picture about what it exists for.

> OpenStack is a large, complex piece of infrastructure software used at scale that many people, even on HN, will never experience directly

IME, openstack is being chopped up and the good parts are moving into Kubernetes. I don't think "openstack" will be a thing in the future, you'll just be using bits and pieces of it. There may come a ship of theseus moment where suddenly openstack is Kubernetes and no one notices

I was originally not going to respond to this, as I don't think this was written in a tone to encourage good discourse; however I wrote something on one of my socials on Friday which is relevant, which I'll paraphrase here.

It's been an extremely long time since I've seen an extreme scale Kubernetes environment which didn't have some OpenStack components, and it's been a long time since I've seen an OpenStack installation which didn't include (at least one) Kubernetes cluster.

And even if this wasn't true... would it matter? We're all working towards a vision of open infrastructure, trying to serve the infrastructure needs of developers who are counting on us. Why do we have to be so tribal about it? It doesn't matter if you're using OpenStack or Kubernetes to automate your infrastructure; it matters that your infrastructure is automated and that you automated it with open source software.

In the end, it's all better when we cooperate. That's happening all over the place! Metal3.io is bringing the power of OpenStack Ironic to an API design more familiar to the CNCF/K8s community. The OpenStack Magnum community continues to make installing Kubernetes clusters on top of OpenStack easier.

Tribalism is just another form of taking your eye off the ball. Open infrastructure is winning; we can't lose focus now!

One of the most interesting parts of OpenDev is that the infrastructure that builds and deploys all the OpenDev services is open.

Anyone can propose changes to practically all of the infrastructure via the https://opendev.org/opendev/system-config project.

Anyone can look through all of the changes; https://review.opendev.org/q/project:opendev/system-config+s...

Every change gets run through an extremely thorough CI system that tests the change and reports results, e.g. https://zuul.opendev.org/t/openstack/buildset/50ce144851224b...

Most of these CI jobs do things like apply your proposed change, deploy the service, then connect up a headless client and take screenshots of the results so you can confirm the correct behaviour of your change; e.g. http://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b62... is from a job that was modifying the gita deployment.

Humans approve changes for merge with Zuul, but Zuul commits the code. When Zuul merges the change, another set of jobs will push things to the production hosts automatically. There is a talk on the overall process at https://www.youtube.com/watch?v=apLHQ4DkIHU

There are obviously bastion hosts and private components to the production deployment, but as much as practical is completely open. There have been contributors with no special access that have developed the deployment of infrastructure services and admins helped only at the end committing some secrets and providing production hardware resources. More commonly, people who have CI issues can jump in and fix their own problems, especially relating to things specific to them (e.g. mirror setup on CI nodes, software versions used, etc.)

One cool thing is that the Zuul jobs publish their production deployment logs in public, but encrypted (these might contain secrets, so are not made open by default). If you work on a particular system, you can request to commit your public key to the service you like and access all the deployment logs. For example, say you are interested in maintaining codesearch.opendev.org. The "infra-prod-codesearch" job deploys changes relating to this service. You can look at all the jobs at https://zuul.opendev.org/t/openstack/builds?job_name=infra-p..., and every job has an artifact that lets you download the logs; e.g. https://zuul.opendev.org/t/openstack/build/c76ec695d19a4e9e9.... Details are at https://docs.opendev.org/opendev/system-config/latest/open-i...

Wikimedia is the only other platform I've seen with a comparable commitment to open infrastructure (unsurprisingly there has been collaboration between the two over the years)

As years of openstacker, happy to see opendev here!

Reply on news.ycombinator.com