Hacker News Reader

Comment by gwking | Hacker News Reader

The last paragraph says: > At some point your docs will outgrow a single folder, and then all bets are off. You’ll want a separate repo with its own build process...

My question is, why is this taken as a given? Is it so hard to have docs and code live together in version control after a certain scale? If so, what is the specific problem and what is the cause?

I ask because I've never been that satisfied with the various ways I've tried to organize projects in git. Recently I've been trying to keep the source, tests and docs together in the same tree so that changes are more localized. It seems to be helping me keep track of things, especially with coding agents so eager to make changes all over the place. I find their proclivity to repeat the same idea in multiple locations (agent instructions, docs, docstrings, help strings, comments) especially problematic.

View on HN

On a large project you will have problems. You can maintain a monorepo anyway as many people do, and deal with the problems of a large monorepo. Or you can go to multirepo and deal with the issues of multirepo. Both have been done successfully, and both have significant problems that you need to work with.

Most people advocating a monorepo have never worked on a project large enough to see the issues with a monorepo and so are arguing for a monorepo without understanding the problems with them. For most people a monorepo is the correct answer because their project is small.

Seems like if you're small enough, a monorepo is the right way to go because it doesn't matter at that scale, and if you're big enough, you'll have the resources to throw at making monorepos scale.

Mono vs poly at scale needs resources. You have different compromises with each and so the resources go to different places. However there is no clear cut winner despite a few mono repo at scale advocates trying to claim otherwise - they are always completely ignoring the issues with a monorepo setup.

Multirepo at scale also needs resources, there's an enormous amount of work required for version bumping and synchronizing everything. People always completely ignore all those chores. Generally, you have either a massive amount of tech debt or you have one person doing nothing but running around doing all that work for everyone else (works great if all that work seems to magically appear for you). You can also be furiously working at automating all those chores, but that's the same level of effort you'd have to throw at scaling out a monorepo--just different.

That was my point - there are issues with both at scale. Trying to pretend that one is better for everyone is wrong. You just choose your tradeoffs.

Exactly because monorepos have least overhead when they’re small, monorepos generally win because you need to be small for a long while until you get big.

By the time you’re “at scale” (who knows), and all these monorepo at scale problems start to overwhelm, you can switch strategy, because the economy of polyrepos is so obvious by then.

So far, I’ve started a new job a handful of times by collapsing a premature polyrepo strategy: people were not experienced enough to merge two git repos without a common root.

I’ve only once went the other way, and it incurred so much overhead, it decreased developer productivity by some small but not insignificant percentage.

To be clear: I’m not a maximalist. All of my open-source work is exceedingly compartmentalised. My DNS library is separate from my external-dns webhook is separate from my fork of external-dns. They could all live in one repo. But FOSS encourages reusability, commercial software encourages clumping and vendoring.

> By the time you’re “at scale” (who knows), and all these monorepo at scale problems start to overwhelm, you can switch strategy, because the economy of polyrepos is so obvious by then.

Conversely to your experience, I have worked at a handful of places who have a monorepo that has been creaking under its own weight for years, but its structure as a monorepo now underpins the business, and so migration to a polyrepo simply never happens, and developers are now checking out a 50GB repo in its entirety periodically.

I'm on the other side of this problem, with a company that went multi repo for bad reasons (political, not technical) and I would give you serious money if you could solve my problems by just forcing me to check out 50gb every now and then...

Instead I deal with a 30+ repo clusterfuck (technically we have 60+ services, but I only have to run half...) that is held together by hopes and prayers, takes literal hours of actual effort to bring everything up to date on master, and has become a fractured hellscape where people are afraid to leave their tightly constrained silos of service combinations.

Long story short... I will take a bad monorepo over bad multirepo any day of the week.

I suspect that your hours of effort to get everything up to date would exist in a monorevel, too. It would exist in a different form, and so it would be harder to measure, but a large part of the work has to be done either way. It's just that certain parts of the work become very visible.

Yeah, compared with the maintenance effort of running around to 100 different repos, keeping everything in sync and the deps all updated, I'd always take the pain of 50GB checkouts.

It looks like these days a combination of git-scalar, git-lfs, and bazel will work for a small 50GB monorepo like that. The largest cost would be porting over the existing build system to bazel and then integrating it with CI. Once that is done, though, the monorepo will just scale, and all the constant ongoing cost of multirepo sync will be gone.

> developers are now checking out a 50GB repo in its entirety periodically

It would, of course, be easy to say "But that's not the fault of monorepos! Clearly, 50GB is a nonsensical amount of source code, and clearly someone committed something they shouldn't have in the past."

But also, with monorepos, the probability of that happening to the repo you use the most is the cumulative sum of all of its projects, since more people's potential git mistakes now happen in the same place.

For the sake of comparison, nixpkgs is 4.5GB right now. It hosts ~140.000 packages, ~12.000 open PRs, one million commits, yadda. So when you reach ten times that size with, presumably, less traffic, I would consider cleaning up the history.

Polyrepos can definitely work; I use them extensively in open-source. I also wouldn't be doing that without Nix flakes all over to chain them together. And even then, I have constant drift because one repo is pinning an old version of another repo, and once I bump the pin, things break lazily.

The biggest benefit, by far, you get from monorepos, is eager evaluation of your dependencies.

> Most people advocating a monorepo have never worked on a project large enough to see the issues with a monorepo

Really? I feel like most of the stuff I have read advocating monorepos are from people at Google, which is a HUGE monorepo.

Google is an advocate of monorepo. However a lot of people are seem to be regurgitating what google wrote about them, but they are not Google scale and have no idea what the problems Google faces are.

Google also is very much in the yell loudly and ignore anyone who points out the problems of a monorepo.

Maybe it is just because I know a lot of Googlers/ex-Googlers, but almost all of the advocates for monorepos that I have spoken to are basing it from their experience at Google. I won't disagree that they tend to gloss over the extensive tooling they have to make it work, though...

Isn't Microsoft also using a monorepo?

> Is it so hard to have docs and code live together in version control

Managers and others won't touch the repo. (Sometimes it's better the don't...)

We had managers and even non-engineers check-in documentation at Google: each page had an "Edit me" button that spawned an editor in a new tab with a CL(PR) ready. It worked very well.

> I find their proclivity to repeat the same idea in multiple locations (agent instructions, docs, docstrings, help strings, comments) especially problematic.

I have not taken full advantage yet, but every source sub-directory can have its own CLAUDE.md or AGENTS.md file, with instructions for a given directory, whenever something like Claude opens a file in a folder, if there's an appropriate agent doc in the housing folder, it should read / apply its contents when working. Not sure how this happens with Sub-Agents on that note.

I think if you want both, you might as well add a ./docs/ directory, and put your md documents in there however you want, this has the upside of letting you have docs with your code always, as well as letting you link to direct source code files.

Weirdo checking in

I always initialize my projects with a src, and docs, directory, for exactly this reason.

My reasoning is that I shouldn't have to go hunting for the docs for the code, or vice versa.

Do you have only those two dirs at the top level? If so how are you finding it? I tend to have a docs dir at the top level, along with other build stuff

I've had no trouble so far. I'd have no problem with ephemeral build artifacts going either at the root, or in src.

edit: I have had one pain point: devs/managers ask me why I do that, and that I stop. I refuse.

I also put everything in one repo nowadays, no matter the usage, the language etc. All is synced, and all is accessible by my LLM. There are a lot of tools to manage monorepos, and frankly most of the time you don't even need them.

It makes releasing software a lot more complicated.

Not a big deal if you're basically the only developer and handroll the process - but poly repositories make release and dependency management a lot more straightforward to wrangle

Most of the time your repo is so small that you won't run into the problems of a large repo and so you don't need those tools. Don't confuse that for monorepos have no problems when they get large.

How large is large? What does large mean? More than 100GB? 2TB?

You have the wrong measure. Large is number of parts. That is partially source files, partially projects/teams, and partially things that are conceptually not related. Likely other things as well.

It is unlikely GB/TB is ever a measure, though if your repo is that big and some people only need a subset of the repo it would be.

Worked at a place (as a tech writing manager) where the monorepo was several TB and company-issued laptops all had 500 GB hard drives. It was like a rite of passage that new writers or people with new laptops would inevitably not RTFM or learn about sparse checkout and try to clone the entire repo, which was not good.

Nice hazing ritual, I hope that behavior extended to other parts of the organization.

Reply on news.ycombinator.com