Hacker News Reader

Comment by kccqzy | Hacker News Reader

> In an ideal world, a version control system should be strictly internal facing and be able to fall over without impacting live traffic; however the Piper team kept discovering that this wasn’t the case.

This is something that surprised me while I was at Google too. Of course a source code version control system shouldn't be a dependency of production, but it turns out it is just so convenient to go from (1) use your source code system to store configuration so that they are versioned and tracked; (2) have tools to read configuration and apply them automatically, Terraform style; (3) have tools to modify said configuration automatically instead of manually. Voila your source version control system suddenly becomes a NoSQL database.

View on HN

Almost everything anyone ever wants to do with production is put files on a server and start a process. The rest comes down to your tolerance for baroque scaffolding and someone else’s janky shell scripts

Google's source code system isn't only used to store config: some teams also use it to store data like ML models. There's a limit on the max file size, causing all sorts of trouble in the LLM era. So it's a bit like Git LFS, but worse (why have the limit at all?).

> why have the limit at all?

Because it's probably buggy and not tested with commits past a certain file size.

Better question, why no use a more appropriate, instead of convenient, tool for the job?

> Better question, why no use a more appropriate, instead of convenient, tool for the job?

Because ML engineers also want version control and easy history and rollbacks and pull requests with code reviews for model changes.

There is no tool that does that except for the version control systems that programmers use. Other disciplines doesn't seem to care as much about these things.

Edit: And in general the extra storage/compute cost of storing it there is less than the saved work in programmer hours from having version control. It is much less worth adding support for larger files than trying to build an entire separate version control solution. Google already has support for images in the version control system, with image diffs etc, there is no reason not to also add support for these large ML models similarly.

> Because ML engineers also want version control and easy history and rollbacks and pull requests with code reviews for model changes.

That's not an answer so much as an excuse.

> There is no tool that does that except for the version control systems that programmers use.

It wouldn't be much effort at all, especially for Google, to write a simple wrapper that relied on a database or zfs snapshots or anything actually more suitable.

Our team just built it, appearing in ICML 24':

https://www.microsoft.com/en-us/research/uploads/prodnew/202...

There are a number of really good internal tools for ML model management. It'd be really weird to rely on Piper to manage production ML models since this does not provide critical functionalities (monitoring, deployment, dependency management etc etc)

That team has some uncommon testing and deployment requirements, and they don't really run a service, so their use of Piper makes some sense to me. Although I agree it's still very weird. They are trying to migrate right now.

When I was there models weights were pretty much universally stored in colossus, either directly or through some bare bones tooling. I’m surprised the team you were working with had requirements that lead them to using piper directly for that.

We did abuse piper for all sorts of things though, most notably “golden” screenshots for diffing ui code changes across multiple search results pages.

Data in Piper (actually, even in SrcFS…?) is stored forever. So the cost of a “small” amount of data like few gigabytes can add up over time (cost for storage, replication, transmission, etc.)

In addition to the benefits above, by storing storing config in version control is 2 party control, presubmit linting, and easy rollbacks. I suppose the trade off is slower rollouts of config and some delay time in pushing out a config but hey, there's no free lunch :-)

Nothing preventing you from building a versioned configuration system using a database or similar, especially if you build tools to actually change the configuration rather than hand editing text files.

Google has this, IIRC it’s called CDPush

CDPush does versioning, canaried rollouts of... hand edited text files.

I believe CDPush still has some latency involved which would be slower than reading raw config data from a DB.

CDPush does the pushing. It's not a database.

The key to this is machine AND human readable formats. That is why the text format of protobufs is such a good approach: - structured formats - declared schema - the parser to read the file is also a validity checker (at least for data types)

There are tools that do similar with JSON but none are as simple and unobtrusive as protobuf tool chain.

When SOX compliance became a thing (remember Sarbannes-Oxley?), many moons ago, I was able to send EY consultants away during their audit for Unix controls, just by showing them that all our (Autodesk Unix team) configs were in SVN, logged, with authors and full change control. The Windows team had to spend months figuring out something that was not even remotely close to control or auditability. Frankly, I just assumed that it was the std practice today and I am surprised people are still not doing it.

Reply on news.ycombinator.com