Hacker News Reader

Comment by mjg59 | Hacker News Reader

> If you have the source, it shouldn't be difficult to build binaries, should it?

You'd think, but no - there's no guarantee that the shipped binary packages can be built with the shipped toolchain or shipped dependencies.

> If you are building a new Linux distribution based on Ubuntu, I don't even understand why you would want to reuse binaries they built.

Because it's entirely unnecessary? Building the entire archive is a huge amount of effort.

> This post would be more meaningful if it literally quoted from unfavorable license terms used by Ubuntu to actually prevent people from making derivatives. Saying that Canonical "appears to require" something is using weasel words.

The terms are at http://www.ubuntu.com/legal/terms-and-policies/intellectual-... - I probably should have linked them, I've just been writing about this enough lately that it's easy to forget that people might read a single post without context.

View on HN

> there's no guarantee that the shipped binary packages can be built with the shipped toolchain or shipped dependencies.

That problem will be with us as long as we use binaries as the primary means of distribution. If you consider that to be a problem, you could put effort into distributions where the packaging is the source code. I know I do.

> Because it's entirely unnecessary? Building the entire archive is a huge amount of effort.

How do you know that the source code provided actually corresponds to the binaries unless you compile them yourself[ with a toolchain you compiled yourself [...]]?

> The terms are at http://www.ubuntu.com/legal/terms-and-policies/intellectual-.... - I probably should have linked them, I've just been writing about this enough lately that it's easy to forget that people might read a single post without context.

No company's policies are perfect. You might be more effective trying to deal with the overall distribution of how software is done rather than imperfections in community leaders. There is a point of diminishing returns in focusing on a small subset of the companies out there while completely ignoring the rest.

Reproducibility from source and build env are fundamental to science or making changes to improve something. Without these, it can appear to signal hoarded proprietariness waving the banner of FLOSS or just a poorly-maintained project without high standards, even if it were unintentional.

> Reproducibility from source and build env are fundamental to science or making changes to improve something. Without these, it can appear to signal hoarded proprietariness waving the banner of FLOSS or just a poorly-maintained project without high standards, even if it were unintentional.

I am typing this on a system running Gentoo that has only 3 proprietary packages installed, with one being Intel's microcode. In time, I hope to lower that number to zero. If you are concerned about this and do not run a source-based distribution, I suggest that you switch.

Source-based distributions are not the only solution. >> https://wiki.debian.org/ReproducibleBuilds

That requires making the builds easy to do, such that Matthew's claim that people cannot easily build things themselves probably is not valid. At least not if that project succeeds.

Anyway, I am happy to see that building from source is becoming easier for users of Debian and presumably Ubuntu by extension.

> If you consider that to be a problem

I do consider it to be a problem, but since reality is problematic I think policies that force people to rebuild all binaries before they can redistribute them are harmful.

> How do you know that the source code provided actually corresponds to the binaries unless you compile them yourself[ with a toolchain you compiled yourself [...]]?

I don't, but I also don't have a verifiable path to bootstrapping a toolchain. I trust that Canonical haven't backdoored their packages.

> You might be more effective trying to deal with the overall distribution of how software is done rather than imperfections in community leaders.

I'm interested in it being straightforward for people to produce modified derivatives of the market leader, ie Ubuntu. I can achieve that in two ways:

1) Helping convince Canonical to change their IP policy 2) Helping convince Canonical to rearchitect their entire distribution infrastructure

(1) strikes me as being easier, so that's what I've chosen to work on.

> I don't, but I also don't have a verifiable path to bootstrapping a toolchain. I trust that Canonical haven't backdoored their packages.

You do not need a verifiable path to bootstrapping a toolchain to reduce surface area of attack to just the toolchain.

That being said, you could try building Clang with GCC and then GCC with Clang (or vice versa) to break any exploits designed to persist in either one indefinitely unless you are the victim of a really special hypothetical attack that targets both simultaneously, although I imagine you could interleave multiple GCC and Clang versions to make such an attack even harder to pull off or even add more compilers to the mix like EKOPath. That should reduce the attack surface area to binutils.

And this would still only buy me something if Canonical are backdooring their packages, which I don't think they are. It's a huge amount of extra effort for what is (as far as I'm concerned) zero benefit.

That assumes that anyone interested in putting backdoors into Canonical's packages never gets a job making their packages. If you are really concerned having trustworthy systems, you need to build from source and if you doing that anyway, you might as well just use the binaries built from source.

If you are looking at some repository for Ubuntu's source code, and find that their build process isn't perfectly documented as regards the version of gcc they use, that really isn't a GPL violation no matter how hard you try to claim that it is.

You don't have a moral right to base a distribution on binaries produced by some other distribution. Unless you are going to produce actual evidence of a legal violation you really don't have anything here.

Not sure whether this was aimed at me, but I don't believe that having source be difficult to build is a GPL violation.

> You'd think, but no - there's no guarantee that the shipped binary packages can be built with the shipped toolchain or shipped dependencies.

If the shipped source doesn't contain enough information to actually perform a build surely that's a straight-up GPL violation (and a far more serious one than any possible ZFS issue, and one that you're in a position to do something about as a copyright holder). If the shipped source doesn't include whatever their developers actually use to build (and I refuse to believe an organization like Ubuntu wouldn't have a unified "build it all" script, or at least a README that described the steps you needed to take) then how can it possibly be the preferred form for making modifications?

>If the shipped source doesn't contain enough information to actually perform a build surely that's a straight-up GPL violation

Why would it be? Build and testing systems for something like a Linux distribution are far more complex systems than a simple script. In any case, there's no requirement in the GPL (or any other OSI-approved license AFAIK) to include components outside of the software itself to simplify building and installing. Of course, there are a variety of things like package managers that do simplify the process but there's no requirement in the GPL that it be straightforward to install from sources.

The requirement is to ship the source in the preferred form for making modifications. That doesn't include standardized external tools, but it surely includes the build instructions (with things like which versions of the tools to use) and test suite (and indeed git history - mjg59 has disagreed with me regarding that before, but I don't think he gave a rationale) - those are things you'd definitely want to have for making modifications.

There are scripts for building each package individually, and these scripts list other packages that are required for the build process. There is nothing to enforce the requirement that the build requirements are a subset of the packages built, or even correlated to them.

> If the shipped source doesn't contain enough information to actually perform a build surely that's a straight-up GPL violation

Package A's source code contains some undefined behaviour that gcc 4.4 tolerates. The distribution upgrades to gcc 5.1 and the build now breaks, but nobody notices because no new version of Package A has been uploaded and so no new build has been performed. Is your position that it was in compliance with the GPL before the gcc transition, and in violation afterwards?

No. If the package A developers when working on the code just use whatever version of gcc's installed and don't have any process then they're compliant (but dumb). But if they have a script, or even just a "building.txt" that says "install gcc 4.4 and set these environment variables, build is broken on 5.x, known issue", and they exclude that from their source distribution then they're in violation. They need to ship the preferred form for making modifications i.e. what they would use themselves or give to a new developer on their team. It's no different from the example of C source generated by a perl script (you can't ship the generated source, you have to ship the original script), and it's exactly what's necessary for the four freedoms - end users need the same facilities that the original developers have when working on the code, not a watered-down version.

> But if they have a script, or even just a "building.txt" that says "install gcc 4.4 and set these environment variables, build is broken on 5.x, known issue", and they exclude that from their source distribution then they're in violation.

They don't.

Really? They being Canonical? They just build their binaries with whatever happens to be installed on Bob's machine this week and then that becomes their official release?

They build their packages with whatever is in the distribution at the time, yes. It happens on the build daemons rather than on the developer machine.

Then how can you get into the situation that the shipped binary packages can't be built with the shipped toolchain or shipped dependencies - isn't that what the build daemons do, use the system to build itself? I mean if they use the previous release or something that's fine too. Whichever way they do it, surely it's documented internally, because it's the kind of thing developers would need to know when developing. I can't believe it would be a case of "if the build doesn't build ssh into the build daemons and update them until it does".

I mean a developer working at Canonical, trying to make a change to one of the packages they're preparing for distribution, has to have some way to answer the question "what version of gcc is being used to build this package". Whether that information is kept in the source tree, on their wiki, or on a post-it on the employee fridge - in any case, it's part of the source in the preferred form for making modifications, because it's, well, part of what the employees use to make modifications.

Then how can you get into the situation that the shipped binary packages can't be built with the shipped toolchain or shipped dependencies

I think that Matthew might be suggesting that not all packages are rebuilt with every release? A binary package might be compiled with GCC-4, then the distribution is upgraded to GCC-5, dropping GCC-4. If for some reason the package is not compatible with the new compiler, you would then be in a situation where the binary cannot be recreated from source without outside tools. Do you know if Canonical recompiles everything on each update to the toolchain?

I have no specific knowledge about Canonical's processes. I'm just amazed that a serious software company in 2016 wouldn't have a reproducible build process for their primary product.

> Then how can you get into the situation that the shipped binary packages can't be built with the shipped toolchain or shipped dependencies

The distribution isn't rebuilt every cycle. If no new source release has been uploaded, the existing binary will be used for the next release.

So if they need to apply a small patch to a given package they'll sometimes discover that it was built 3 versions ago and no longer compiles? Yuck.

mjg59: when you link to a Shuttleworth comment as a 'flat refusal', and that comment is actually a statement of position and procedure to achieve your general aims (if not your specific ones), but finally ends with a paragraph stating that you misrepresent what Shuttleworth says, it's hard not to believe him.

Reply on news.ycombinator.com