How do people manage hundreds of Git repositories? | Hacker News Reader

How do people manage hundreds of Git repositories?

17 pointsSevii15 comments

I have worked with several large companies using git as their version control system. One hosts several hundred git repositories in bitbucket stash, the other has around a hundred repositories in github. We recently ran into an issue with joda-time where upgrading past java8u60 caused apps to fail on startup. So my team is tasked with getting all of our microservices working with the latest jdk.

What tools are there to modify and commit similar changes across 100+ repositories at once?

I have found gitwalk https://www.npmjs.com/package/gitwalk which does a subset of what I want focused on github. But google has not revealed many other tools that can do discovery and transformations across 100s of repositories.

View on HN

This looks like a one-off thing you need to do semi-manually and methodically.

Every step of the way you need to balance the effort of doing something with a script vs doing it manually.

How long would it take to manually make the change in all 100+ repos... vs. how long would it take to write a script to do it. Would the script be robust enough to properly make the change in like 90% of them... etc.

These are things you need to ask yourself in your endeavor.

You'd want to do all this work on a new similarly named branch (or branches depending on your workflow) for every repo.

Perhaps write a script to validate changes to all 100+ repos on that branch. Then when all the work is done... do a massive merge.

It probably depends on exactly what and how complex your “transformation” is. You definitely can do lots of automation with $ git itself in terms of committing, merging, opening PRs on GitHub via $ hub, etc.

Another idea is if you can refactor the common bits into a shared standard library, for instance the design pattern is something like a system with a bunch of plugins, then most of the plugin repos become very tiny. Ideally you’d also have some sort of infrastructure automation around deployment.

We have shared libraries and maven plugins setup. This is a situation where I really wish we used a monorepo and I could just do one commit across 100 microservices using grep and other tools to update pom.xml files. I have read whitepapers from google on how they support this kind of operation, but I need it for many git repos.

This sounds like StackOverflow question. Anyway, if you're proficient in Java, why don't you write small app which will iterate over all Git repo dirs, modifies the files, makes commit, push and creates PRs? Not a big deal, right?

That is the current plan, combined with plenty of manual work verifying jenkins jobs and deployments.

If you can do it if there was just one repo, then juat clone those 100 repos under the same parent directory and execute the script in the parent directory

and then in a loop commit the changes

I haven't found anything off-the-shelf that helps. I manage a project with > 50k repositories in a GitHub organization. For the past few years I've been developing software that helps with my [particular goals](https://github.com/gitenberg-dev/gitberg), but I'm afraid there isn't much there that may help you.

Have you looked at a tool like: https://github.com/mixu/gr

Going to look at it, thanks.

It's too late for you, but I think that it's useful for other folks to see your example and realise why multiple repos are a bad idea.

In the general case, each organisation should have a single repo containing all of its code. There are times to violate this rule, but unless you're in extraordinary circumstances, more than a dozen repos in an organisation is an extremely bad mistake.

Can you elaborate on this a bit? What are the advantages to keeping everything in a single repo for the organization vs. a different repo per project?

Basically, having multiple repos means that one person can change a dependency of someone else's code without fixing the dependent code. This is unavoidable with independent organisations, but is easily avoidable within an organisation which uses unit & integration testing within a single repo (the use of a single repo means that a suite of changes may be applied atomically).

In general, a process which allows one person in an organisation to stymie another's progress is a broken system.

Grep + sed or replace in Files for the changes. And gitall to commit them all. There is no need to get fancy if simple tools will do the job.

`mr` is good for keeping track of, and working with, many repos in bulk.

http://myrepos.branchable.com/

IIRC it has a good man page explaining more advanced use.

Reply on news.ycombinator.com