Ps thanks for getting scipy into third_party all those years ago.
It's worth noting that this is only viable at Google because they don't use git. Git's insistence on every client having a full copy of all history of every file in the repository makes monorepo much more expensive.
I see conflicting reports over whether google use Perforce or something proprietary called "piper"?
Note that there's nothing forbidding you from writing a virtual git filesystem that fetches objects from some centralized repo as files are open()ed. Git on cloud steroids.
You might be happy to know Microsoft created a Virtual File System for Git[1] [2] so you do not have to have every file checked out in your working directory. Microsoft uses Git in a monorepo (for Windows, and it's 2.5 million files/300GB[2])
(It may still be backwards-compatible, but it's been years since they turned off the last real Perforce and I haven't worked there for years myself. So it may have diverged.)
The underlying assumption it that the project will fold in 2 years anyways, so one may get away with never doing dependency upgrade.
The idea of wanting to make a change to a third party library and then see all the downstream consumers who would be broken by that change if they updated to start consuming that change is an incredibly stupid thing to want, and it’s no measure of success whatsoever to build something that gives you that information.
It’s like the most giant case of coupling you can imagine (letting the statuses of thousands of consumer apps act as any type of constraint on the developer choices of the third party library, as opposed to all those consumer apps opting in to changes on their own terms by updating their dependencies).
Imagine if I have shared a bunch of copies of my resume with a bunch of recruiters. They are out there selling me as a candidate or whatever. Now I decide I want to change my resume, but I don’t know if it’s going to upset the approach some recruiter is taking.
If I can’t update my resume unless I first consult a big oracle that tells me which recruiters will be negatively impacted, that’s a problem, and not at all some type of live-with-able “customer service” positive thing. It’s just plain old bad coupling.
Creating such a system that could automatically diff the old resume’s usage constraints against my proposed changes would be a gigantic waste of time. The exact opposite of something to celebrate.
I say this as someone who routinely writes in-house software libraries used by dozens or hundreds of other apps, various teams, and even a few that are open source.
The primary thing gauging the health of our development is that we are decoupled from any consumers. We are free to make whatever changes we want, and whether downstream teams would like to receive those changes is wholly an opt-in process with versioned dependencies and easy rollbacks controlled by those consumers.
It's also useful information in the edge case where consumers are relying on undocumented or unintentional behavior in your package.
Yes, you don't want a hard-constraint of no-breaking changes ever, but knowing immediately when a change is breaking change (especially if you didn't intend it to be) is useful.
> “in practice its important to remember that your package only exists to be consumed by its dependencies. That is its sole purpose. If your changes aren't serving those consumers, then they're the wrong changes.”
I agree completely and that’s exactly why you want downstream consumers to opt-in to your changes.
As the library writer / maintainer, nobody knows better than you how to implement the behaviors downstream consumers want. Sure, those other folks know what they want, but are not at all a trustworthy signal for how to solve it for them.
If you are constrained by what breakages your new approach would introduce, this is backwards, exactly from the “in practice” perspective you described. That means you are not able to actually solve your consumers’ problems, create new solutions, refactor old bad ways of working, because you are coupling the what with the how.
The fact that you are beholden to your consumers is all the more reason to decouple the development process from the delivery process. It only makes this idea of wanting a big oracle to tell you what would be broken not because of functional incorrectness but because of a consumers lack of accommodating the new changes all the more egregious.