HNHacker News
TopNewBestAskShowJobs

thaJeztah

91 karma · joined December 28, 2013

[ my public key: https://keybase.io/thajeztah; my proof: https://keybase.io/thajeztah/sigs/M7UVXzHD5P6-_aoQ0z4G4FV8C5ctDoLuZ0Tpn8TlY2U ]
submissionscomments
thaJeztah··on Docker 29 has changed its default image store for new installs
Yes, compression being part of the OCI image's digest was (in hindsight) a poor decision. _Technically_ OCI images allow uncompressed layers, and the layers could be included without compression (and transport compression to be used); this would allow layers to be fully reproducible. We explored some options to do this (and made some preparations; https://github.com/containerd/containerd/pull/8166), but also discovered that various implementations of registry clients didn't handle transport-compression correctly (https://github.com/distribution/distribution/pull/3754), which could result in client either pulling the full, uncompressed, content, or image validation failing.
thaJeztah··on Docker 29 has changed its default image store for new installs
I'm a maintainer of the Moby project (which is used to build the Docker Engine), and saw this post got some attention, so let me try to outline some of the changes and motivation. Happy to answer questions if there's any.

First of all, some history; the Docker Engine was a monolith daemon that provided many services; this worked well when using Docker as a standalone solution, but when used as runtime for Kubernetes, this wasn't ideal; many components were not designed for this purpose, which meant they had to be replaced / overridden with hacks to make it work. The containerd project was created to provide a more modular runtime for the container ecosystem, providing separate subcomponents (a containerd runtime, image/content storage) for the container ecosystem to build on. It was created "from scratch" with lessons learned over the Years, providing a modern foundation.

While docker has used containerd as a runtime for many Years, it still used its own implementation for storing images ("graph-drivers"); this implementation started to show its age and had many limitations; graph-drivers have no native support for multi-platform ("multi-arch") images, no support for OCI Artifacts, and no reproducible images when pushing to different registries (among others).

Around 4 Years ago, we started to re-implement the image storage using containerd "snapshotters"; our initial goal was to provide a mostly seamless transition; add multi-arch support, but keep the UX as close as possible to the graph-drivers. Around 2 Years ago, Docker Desktop changed to using the containerd image storage (snapshotters) as a default for new installations, and Docker v29 made it the default for Linux installations.

While we kept most of the UX similar, there are some differences; when storing an image with graph-drivers, docker would pull the OCI image, extract the content (layers), and discard the (compressed) layers. While this reduced storage, it also made images non-reproducible as the image had to be re-constructed when pushing to a registry (which also resulted in slower pushes).

The containerd image storage uses a different design, where a copy of the compressed artifacts are preserved (by default); this requires more storage to keep these extra blobs, but reduces duplication and increases push performance. It was the decision containerd maintainers made early in their design process, and all containerd-based tools have used this model since the start of the containerd project.

We have a couple of roadmap items to improve this in future; some are outlined in this ticket; https://github.com/moby/moby/issues/51581, but there's other options that will become availeble through the containerd image store; support for erofs as an alternative to (tar) compressed image layers, as well as automatic garbage-collection (which would reduce the need for manually pruning content through `docker system prune` (and related commands).

(FWIW; docker still provides graph-drivers as an alternative https://docs.docker.com/engine/storage/drivers/select-storag...)

thaJeztah··on Please don't upgrade Docker without asking first
I wrote that comment at the time; to add some context to that comment: removal of the "login to download" was shortly after the "Docker Enterprise" business went to Mirantis. While Docker started as a developer centric tool, focus shifted to Enterprise products (Docker Enterprise Engine, UCP, Docker Trusted Registry, Docker Desktop Enterprise). After the move of the enterprise products to Mirantis, Docker's focus went back to developer products.
thaJeztah··on Running Docker on Apple Silicon M1 (Follow-Up)
And Dave decided to end the week with a bang; here's Docker Desktop running on an M1 (with lots of Duct tape); https://twitter.com/mugofsoup/status/1332382741892124675?s=2...
thaJeztah··on Docker Enterprise Edition
> I'm aware of runC but don't know if Docker images are realistically portabl

(It wasn't clear from your comment if you were aware of this) Since last year (docker 1.11) docker itself no longer is a runtime, and uses runC as the default runtime (https://blog.docker.com/2016/04/docker-engine-1-11-runc/)

Additional OCI compliant runtimes can be configured on the daemon (https://docs.docker.com/engine/reference/commandline/dockerd...), and can be selected per container, using the "--runtime" option on "docker run" (https://docs.docker.com/engine/reference/commandline/run/#op...)

thaJeztah··on Docker Enterprise Edition
The quarterly ("stable channel") CE releases (17.03, 17.06 and so on) are supported for 4 months, and will not get new features during that period. EE quarterly releases have a 1 year support period, and also won't get new features.

During the support period, bug fixes will get back ported to those versions and released as "patch" releases (e.g. 17.03.1).

When installing, you can choose to install either the "stable" (quarterly) channel, or the "edge" (monthly) channel.

thaJeztah··on Docker in Production: A History of Failure
A new "docker prune" command is added to docker 1.13 to address this, see https://github.com/docker/docker/pull/26108
thaJeztah··on Docker was unavailable in Ubuntu/Debian repos
Hi! I work at Docker as well; and happy to tell that we found the cause of this issue, and managed to resolve it.

Be sure to clean your apt cache before trying again (apt-get clean && apt-get update) if you're still seeing this.

thaJeztah··on An OCaml/Mirage-friendly implementation of the Plan9 filesystem protocol
Bottom of the README says:

> This library supports the 9P2000.u extension

thaJeztah··on This Industry is Fucked
Thanks so much for writing this up. Happy to help getting those shirts out. Let me know if there's any way I can help out, with the shirts, or any other way. This just has to stop.
thaJeztah··on Docker Image Insecurity
Not disputing that the "verified" message is misleading, but the comment you referred to was about "not yet being able to create a signed image".

I don't think there's anything yet in the docs with regard to signed image, apart for the release notes[1] mentioning it being a "sneak peak" of a coming feature, that is under development

[1]: https://docs.docker.com/v1.3/release-notes/#new-features

thaJeztah··on Docker Image Insecurity
Sounds interesting. Perhaps you should create a proposal for that on the docker issue tracker, so that it can be discussed as a possible alternative?
thaJeztah··on Announcing Docker Machine, Swarm, and Compose for Orchestrating Distributed Apps
> because the command is "docker run", with additional settings

The `-m` flag was already present in docker run. Swarm makes use of that flag to decide on which host to put a container. The `constraint:` setting is indeed new, but (afaik) the values behind it are not pre-defined, so if someone develops an alternative clustering-backend they should be able to make use of that for their needs.

In any case, Swarm will be pluggable, so other back ends are possible and can be developed by other parties.

thaJeztah··on Announcing Docker Machine, Swarm, and Compose for Orchestrating Distributed Apps
> confused over how many components and services you have to contend with now

You don't have to. It really depends on what you're trying to do. Using Docker alone, you'll be able to build and run containers, link them together to build a "stack" etc.

If you want to make building a stack (a group of containers that together form your application) easier, you can use an orchestration tool to automate this, for example, Fig, Crane, or now Compose. Or, create a bash script to do this; it's up to you.

If you want to build a cluster (run your containers distributed over several servers), you can do that with docker alone, but it will get hard to manage. You can build a tool for that (making use of the Docker API), or use an existing tool, like Flocker, Shipyard or now Swarm.

So if all you need is running a few containers on a single host, Docker alone may be enough for you, in that case you can safely ignore the other stuff for now.

thaJeztah··on Announcing Docker Machine, Swarm, and Compose for Orchestrating Distributed Apps
> For comparison, Quay.io was able to do it with just a few engineers.

To me, that actually shows that you don't have to rely on Docker Inc online services and are free to choose other options (that perhaps are better?)

thaJeztah··on 20 Freescale Employees were Confirmed Missing on Malaysia Airlines Flight MH370
Exactly my thoughts! No matter what the cause is of this disaster, we're talking about human lives here. Please keep this in mind.
thaJeztah··on Arduboy: The Interactive Digital Business Card
Nice!
thaJeztah··on List of possible changes, updates, additions for PHP 6
+1 if debugging is present out-of-the-box, even first-time users will have no excuse not to use proper debugging.

While installing XDebug may be no rocket-science, I've encountered too many situations that it just didn't work reliably, causing developers to revert to nasty var_dumps() for debugging.

thaJeztah··on List of possible changes, updates, additions for PHP 6
I agree completely, maybe PHP should use the same approach as the jQuery migration-plugin, which;

- Logs usage of old or (to-be) deprecated functionality

- Provides deprecated functionality so that existing code doesn't break

This way;

- The PHP API can finally be cleaned up (please, also move those old functions to a separate section in the documentation)

- Older code can still run, but enabling the 'migration/compatibility' plug-in (should be possible to enable/disable this per site, not for the whole server at once)

- Developers are informed that their code uses deprecated functionality, giving them time to migrate code.

Although this approach is comparable to the E_STRICT flag, it is going one step further, in that the deprecated functions are actually removed from the core, and must be manually enabled/added via an optional plugin.

Reorganising the documentation is an important part of this; cleaning up the old stuff will make PHP less confusing (should I use 'implode()' or 'join()'?), and will silently guide new developers away from the old stuff.

On a final note; if this approach allows hosting-providers to upgrade PHP without breaking their client's code, adoption rate of newer PHP versions will probably be faster as well!

thaJeztah··on Important Kickstarter Security Notice
1Password also has a 'history' for previous passwords
thaJeztah··on Windows 8.1 forces the user to interact in a way that doesn’t work
And, apparently the official way to do things ;)

"your computer will restart in 10 minutes", no you cannot finish your presentation, or keep to your deadline. We will forcibly restart your PC and don't allow you to save your work when the countdown has completed... This update solves a potential data-loss probl.........

One customer less. Problem solved!

thaJeztah··on Windows 8.1 forces the user to interact in a way that doesn’t work
I agree that you'll be able to get used to almost anything, however;

The old location (inside the start menu) was at least a 'logical' location, because the start menu in W95 was designed to be the central place for anything you did. The only confusing part was the naming.

Settings is possibly the worst possible location to put this; - In daily use, you will never use the settings menu - Settings are meant to make a (permanent) change in the configuration of your computer. (I configure my computer to 'reboot'? Should it reboot whenever I switch it on?)

thaJeztah··on De La Soul to Make Entire Catalog Available for Free
Yup, same here..
thaJeztah··on Important security information for FRITZ!Box users
Aparently, this is not just an advisory, but this issue has already been used 'in the wild'. Hackers have been able to make VOIP calls through users modems.

Even worse, AVM decided to store password only encrypted (and, as turned out, easily de-cryptable), not hashed! I am very surprised that such a well-known manufacturer took security so lightly. Storing password almost in plain text in 2014 is just not tolerable.

Class action lawsuit anyone?

thaJeztah··on Making PHP Safer
A good starting point would be to implement type-hinting for a generic 'scalar', i.e.;

    mymethod(scalar $foo)
This simple type-hint would already solve a lot of problems where non-scalar values cause havoc (least of all, automatically casting of arrays to 'Array' as string and other cases, like trying to use an array as key in an array).

I've seen a proposal to use a generic 'numeric' type-hint. I'm not sure this is a very good idea, really. If it handles values in the same way as PHP's is_numeric(), this will lead to confusing situations; "0xA", after all is a numeric value. To allow values like this, it's better to specify 'scalar' as type-hint and handle conversion/casting manually.

I personally don't like the fuzzy '~int' (int-'ish') type-hint, it simply doesn't look very clean (looks like some kind of a RegEx). Besides, how should a method pick the best way to cast a non-integer value? (Again, is "0xA" a typo by the user, or does it represent a hex?).

The only situation where automatic casting/fuzzy matching may be suitable, is for Object that implement a specific interface (e.g. ArrayAccess interface).

By lack of real polymorphism in PHP, I'd opt for a more explicit approach, while keeyping flexibility by allowing multiple types to be specified, similar to the PhpDoc notation. Something like;

1. The 'classic' approach

    // Accepts any type
    myClassicMethod($foo)

2. Explicitly 'classic' approach

    // Explicitly accepts any type
    myClassicMethod(mixed $foo)

    // Effectively, this results in the same behavior as 1.,
    // however, it clearly shows that 'mixed' values are accepted
    // by-design and not because of omitance
3. Strict-ish approach

    // Only accepts scalar values
    myClassicMethod(scalar $foo)

4. The 'polymorphic' approach

    // Accepts 'int' or 'string', but NOT 'bool', 'float' an non-scalar types
    myFuzzyMethod(int|string $foo)
    {
        // Further type-casting and checking
    }

5. The 'strict' approach

    // Only accept 'int' values, NO 'int-ish' values
    myStrictMethod(int $foo)
    
    // The code *calling* the method is responsible for converting
    // arguments to the right type, using any method nescessary 
    // for the situation
    
    echo myStrictMethod(intval($foo));

    // or, maybe a simple cast is sufficient;
    echo myStrictMethod((int)$foo);
    
    
In all approaches above, handling invalid arguments (out-of-range etc.) should be the responsibility of the method.

Having said that, unless the method's actual purpose is to convert/cast types (e.g. convertToBool(int|string $value)), the last approach (strict) offers the clearest separation of responsibility (again, IMO);

In many cases, a method cannot be held responsible for picking the best approach to convert values, simply because it does not know where an argument originates, what it should tollerate and how conversion should be handled.

Take for example, a web-form containing a 'currency' input. To offer a user-friendly experience, in many cases it is preferable to be 'forgiving' when handling 'faulty', but (sometimes) trivial user input, like;

- User includes currency-symbol in the field (€ 123,45)

- User uses incorrect decimal-separator for the website's locale (123.45 in stead of 123,45)

Conversions like this are part of the 'business logic' of the application and cannot be reliably performed automatically by the type-hinted method without causing unexpected results.

Type hinting should just do that; 'hint' what is expected and 'throw' if this expectation is not met.

I would really oppose to too many 'fuzzy' matches. Automatic conversion between types (the infamous PHP type-jugling) is a major reason for many headaches while developing with PHP. Disguising them as 'type-hints' if only fooling ourselves.

I would like an option to define custom type-casting handlers to have more control over the way PHP will 'juggle' values and to handle specific situations.

thaJeztah··on Making PHP Safer
Sounds very promising, will definitely give this a spin. Hopefully something like this will make its way into PHP by default.
thaJeztah··on Happy new year and goodbye bzip2
XZ sounds interesting. Anybody know if work is in progress to add XZ support in webbrowsers as an alternative to gz compression? Any webserver already supporting xz?