Hacker News Reader

Comment by candiddevmike | Hacker News Reader

I wish containers hadn't created the abstractions of a registry and an image instead of exposing it all as tar files (which is what it kind of is under the covers) served over a glorified file server. This leads to people assuming there's some kind of magic happening and that the entire process is very arcane, when in reality it's just unpacking tar files.

If you want to DIY a container with unix tools, this should help: https://containers.gitbook.io/build-containers-the-hard-way/

View on HN

A .tar file on a file server wouldn't suffice, because you have to store a bunch of metadata for that .tar file. But now you could argue that the metadata could just be a file next to the tar file. True, but container registries are kind of like that + optimization + some useful extra features. If you were to start from a simple file server and think about the stuff you need, then after some iterations you would actually end up with a container registry.

Docker save (and presumably podman too) adds that metadata as a file in a tar. Each layer becomes a tar that is nested in the top level tar.

Indeed, registries don't do much more than what:

    docker image save foo:latest \
      | ssh pod docker image load

does (the stream is a docker tar archive with layers and metadata).

A container registry allows you to download multiple layer simultaneously and only those that you need (that you don't have in cache).

AFAIK this is how it worked with classical OS packages (think rpm/yum/apt/etc.)

Packages and metadata was all prepared for distribution over nothing more than static file hosting and http(s) to leverage as much existing infrastructure as possible.

"Signing Images with Docker Content Trust" explains how cryptographic container image signatures work w/ Docker Notary (TUF) https://docs.docker.com/engine/security/trust/#signing-image...

The TUF spec (and PyPI TUF PEPs) explains why a tar over https (with optional DNSSEC, a CA cert bundle, CRL, OCSP,) isn't sufficient for secure software distribution. "#ZeroTrust DevOps"; #DevSecOps

What's the favorite package format with content signatures, key distribution, a keyring of trusted (authorized) keys, and a cryptographically-signed manifest of per-file hashes, permissions, and extended file attributes? FWIW, ZIP at least does a CRC32.

We now have the Linux Foundation CNCF sigstore for any artifact, including OCI container images.

W3C ld-proofs is a newer web standard that unfortunately all package managers haven't yet migrated to. https://news.ycombinator.com/item?id=29355786

Because ld-proofs is RDF, it works in JSON-LD and you could merge the entire SBOM [1] and e.g. CodeMeta [2] Linked Data metadata for all of the standardized-metadata-documented components in a stack.

[1] https://github.com/google/osv/issues/55

[2] https://github.com/codemeta/codemeta

Isn't this reinventing the wheel somehow? Linux distros have been using tar.gz over HTTP (not S!) and relying on GPG for years.

But most Linux distros are not that great either. They don't implement TUF and are not safe against some vectors prevented by that. It's OK to start anew when container distribution poses a different problem anyway.

https://en.wikipedia.org/wiki/The_Update_Framework_(TUF)

The linked page doesn't explain anything at all. It says some incredibly obvious things (remotes can be compromised and you should be prepared for that) and then goes on to say the subject fixes all that, but there's actually no explanation of how it does that or what it does to get there. It's a marketing piece, essentially.

From this about DNSSEC and software supply chain security a couple weeks ago, from this about Android packages: https://news.ycombinator.com/item?id=27700513 :

>> [...] Sigstore is a free and open Linux Foundation service for asset signatures: https://sigstore.dev/what_is_sigstore/

>> The TUF Overview explains some of the risks of asset signature systems; key compromise, there's one key for everything that we all share and can't log the revocation of in a CT (Certificate Transparency) log distributed like a DLT, https://theupdateframework.io/overview/

>> Certificate Transparency: https://en.wikipedia.org/wiki/Certificate_Transparency

>> Yeah, there's a channel to secure there at that layer of the software supply chain as well.

>> "PEP 480 -- Surviving a Compromise of PyPI: End-to-end signing of packages" (2014-) https://www.python.org/dev/peps/pep-0480/

>>> Proposed is an extension to PEP 458 that adds support for end-to-end signing and the maximum security model. End-to-end signing allows both PyPI and developers to sign for the distributions that are downloaded by clients. The minimum security model proposed by PEP 458 supports continuous delivery of distributions (because they are signed by online keys), but that model does not protect distributions in the event that PyPI is compromised. In the minimum security model, attackers who have compromised the signing keys stored on PyPI Infrastructure may sign for malicious distributions. The maximum security model, described in this PEP, retains the benefits of PEP 458 (e.g., immediate availability of distributions that are uploaded to PyPI), but additionally ensures that end-users are not at risk of installing forged software if PyPI is compromised.

>> One W3C Linked Data way to handle https://schema.org/SoftwareApplication ( https://codemeta.github.io/user-guide/ ) cryptographic signatures of a JSON-LD manifest with per-file and whole package hashes would be with e.g. W3C ld-signatures/ld-proofs and W3C DID (Decentralized Identifiers) or x.509 certs in a CT log.

> FWIU, the Fuschia team is building package signing on top of TUF.

W3C Web Bundles + Linked Data for the SBOM and metadata could be a good solution for software supply chain security in general: https://news.ycombinator.com/item?id=29296573 :

>> Web Bundles, more formally known as Bundled HTTP Exchanges, are part of the Web Packaging proposal.

https://web.dev/web-bundles/

>> HTTP resources in a Web Bundle are indexed by request URLs, and can optionally come with signatures that vouch for the resources. Signatures allow browsers to understand and verify where each resource came from, and treats each as coming from its true origin. This is similar to how Signed HTTP Exchanges, a feature for signing a single HTTP resource, are handled.

How do these newer potential solutions compare to distributing packages and GPG-signed hash manifests over HTTP, with GPG public keys retrieved over HTTPS (HKP)? (Maybe with keys pinned in the GPG source codes for common key servers? Or why not?)

Are DNS (DNSSEC, DoH, DoT downgrade attacks), CA compromise (SPOF), and x.509 cert forgery still the significant potential points of failure? What of that can e.g. Web3 solve for?

Wut? I just make websites man.

I don't want to be dismissive, but I hate deploying my applications for this reason. I'm an application developer. I'm not averse to infrastructure as code, and containerization, and I'm happy to do ops for my preferred stack. But I can't learn all this stuff too.

Conveniently, Docker has abstracted 99% of what she said away from you.

Does `podman trust` also work yet? Shell commands from the first Docker docs link above:

  docker trust key generate jeff
  # docker trust key load key.pem --name jeff

  docker trust signer add --key cert.pem jeff registry.example.com/admin/demo
  docker trust sign registry.example.com/admin/demo:1

  export DOCKER_CONTENT_TRUST=1
  docker push registry.example.com/admin/demo:1


  docker trust inspect --pretty registry.example.com/admin/demo:1

  docker trust revoke registry.example.com/admin/demo:1

I hope the image registry abstraction opens up the possibility to improve the storage and transfer of images without breaking backwards compatibility. See https://www.cyphar.com/blog/post/20190121-ociv2-images-i-tar for why tar is not optimal.

I don't see how without breaking compatibility, as that would change the image hash.

Agree 100%.

Running containers on AWS requires uploading an image to a registry/repository, but there isn't any API to actually upload the .json manifest files and .tar.gz layer files they need, even if I have them sat in a folder in front of me (compare this to Lambda, where we can upload a .zip to S3).

The worst part is that the "official" way to upload artifacts to a repository actually requires installing the 'docker' command, running some sort of "login" command in that, piping around some secret tokens generated by the AWS CLI, etc.

The only other approach I could find is their "low level" chunk-based upload API. I wrote a Python script around that (pretty much just a 'while' loop), so I can avoid this docker sillyness.

https://docs.aws.amazon.com/AmazonECR/latest/userguide/getti...

Looks like Docker's registry API (what the docker CLI uses) supports doing single-chunk uploads: https://docs.docker.com/registry/spec/api/#monolithic-upload.

I think accessing AWS's Docker API is what requires the "fake login dance" to begin with.

Yeah, but "fake login dance" is mostly just a consequence of the Docker CLI's design. As an API client you just get an API credential and use it to authenticate against the API.

I don't mind that registries are a thing but I hate that they're so baked into the tools, which I think is what you're getting at.

If I'm starting a project, I want everything small, simple, and self-contained within my project directory. If, right out of the gate, I'm supposed to already have a daemon running (as root it seemed for a while?) which will do the building and execution, and a "registry" set up somewhere, I've already got pieces I likely don't understand and magic commands and high level abstractions for interacting with them, and introductory material telling me "this magic command is all you have to do! look how much you don't have to worry about!"

But I am worrying about it. And that's why I love when people write these "hard way" guides. So thank you for that link.

Actually it's the process how I (DIY) deploy several static sites from my local machine. I build the Docker image (hugo sites with nginx, expose a single port for HTTP traffic) and save it as tar. Ironically the base images do come from a registry, but I can't deploy to a public registry and don't want to host my own.

On the server where I need to run that site, I just transfer the tar, load the image and run the docker image. It's so straightforward I much more prefer this way than being dependant on external registry sites for deployments.

Why wouldn't you just run a vanilla nginx image and mount in the folder of site content? Then a deploy would be as simple as rsyncing the built folder from your laptop to the server, no need for sending a whole image every time

The docker/OCI container format is incredibly clunky.

.wh. files? S_IFWHT? Never heard of 'er! Layers as tars that need extracting? #gzipallthethings (what is that you're saying? gzip is slow?)

They did add mime types for layers, so in theory you could do something better, but that's not going to happen because it's not backwards compatible.

Reply on news.ycombinator.com