Hacker News Reader

Comment by malikolivier | Hacker News Reader

Once you enumerate what your development pipeline actually depends on, you will see it rely on a lot of stuff. A typical CI run pulls from npm, PyPI, Docker Hub, a distro mirror and a handful of third-party, none under your control, none with an SLA to you. Third-party outages, yanked versions, compromised packages are the things that can waste your time. Some packages may also drop support for whatever distro you're still using, and the mirrors may disappear (e.g. older Python versions, ROS, etc.).

Shameless plug: To fix all those development pipeline issues, I am working on StableBuild, a tool to easily freeze and pin Docker images, operating system packages, Python packages, and arbitrary build dependencies; in 5 lines of code: https://stablebuild.com .

View on HN

> CI run pulls from npm, PyPI, Docker Hub, a distro mirror and a handful of third-party, none under your control, none with an SLA to you.

To be honest, they shouldn't do so directly. For the obvious reason you stated, they don't have a SLA to you or your company. But also because it is a shitty thing to do as a company. Companies should have their own proxies and mirrors for any repository they pull from. Both to be in control of the dependencies you as a company rely on and to not unnecessarily put load on free third party repositories.

Had not heard of your solution. More often than not I come across Harbor, Nexus or things like GCPs internal stuff.

> to be in control of the dependencies

Maybe I'm out of touch, but I think control over dependencies is underrated. It's not just about freezing and pinning. There should be better tooling to support the whole dependency evolution pipeline. For example updates should be reviewed/ingested/integrated/validated based on local policy not based on source release schedule.

This is why some projects will vendor their dependencies - changes within the dependency itself will also be reviewed, and it's easier to locally maintain patches to those dependencies until the patches can be upstreamed.

The problem is, there's a difference between doing that for a handful of C or Go libraries, versus trying to vendor thousands of NPM libraries and all of their interdependencies. So it's very ecosystem dependent.

Agreed. Your self-managed vendoring example is quite near one end of the continuum. I'm suggesting that there might be space for better tooling from there all the way to the NPM end. For example, automated and/or collaborative triage and review of change sets so that you don't see the changes you don't care about, but can easily review and integrate critical fixes.

The fix is easy and ubiquitous. You host your own artifact repository and docker registry. This also makes sure your supply chain remains easily auditable.

That works for artifacts, but there's no easy solutions for all the security tools that relies on pulling up to date external databases. Databases that are out of your control and often down in my experience.

Reply on news.ycombinator.com