Go 1.18 will embed source version information into binaries | Hacker News Reader

Go 1.18 will embed source version information into binaries

utcc.utoronto.ca

208 pointsnnx91 comments

This is really useful. Almost every project I’ve worked on has had to have a build script in part to inject this info so it being a first class feature makes life just that much easier.

I found Nim's gorge command really useful. It runs a command at compile time and stores the output in a variable, so you can simply do things like:

    const compile_version = gorge "git describe --tags --always --dirty"
    const compile_time = gorge "date --rfc-3339=seconds"

That is very nice, thank you for sharing

So simply compiling code could execute any random code? How is that safe?

It is the normal way of the world. Makefiles are arbitrary code execution. 'build.rs' in rust is the same. npm's package.json has an install script.

Often this arbitrary code is to do things like run "pkg-config --libs" or such to find dependencies to link against, or generate some files that shouldn't be checked into source code, but rarely does it have sandboxing or other restrictions.

Languages, like Go, which don't let a package execute arbitrary code on installation are the exception.

What do you normally do with code after you compile it?

It's never safe to compile code, of course it can execute any random code, otherwise it would be useless...

It's true that most languages's build systems (nim, rust, autotools, makefiles, etc) are unsafe to execute if you do not trust them.

Go does stand in contrast to this. `go get` and `go build` cannot execute arbitrary code, and if you use those two commands to build untrusted code, in theory your machine should still remain uncompromised. They release CVEs for any issues here (such as https://github.com/golang/go/issues/29231).

Of course, if you run the code you compiled, that is unsafe, but just compiling it is supposed to be fine.

At the same time, I was thinking how much of internal structure can be exposed. :)

If there are projects where you don't want to reveal this info in the binary, you can turn the feature off with:

-buildinfo=false

Yes, but it is turned on by default. And 99% of people will go and copy-paste existing tutorial code to their projects.

I especially found out this is really useful for extracting information form non-stripped firmware binaries. They lay around on some S3 buckets, where cheap IoT manufacturers think it's safe to expose them online for updating purposes...

Not much:

> the currently checked-out revision and a flag indicating whether edited or untracked files are present

Basically, it's including a git commit ID, which is a hash, and flags that indicate if there are local changes.

It's pretty common to include build hashes in shipping code, logs, ect.

The build version, embedded in the build, should match with a label or tag in source control so that this information should allow you to immediately get everything you need out of source control. In general you don't want more than that embedded in builds.

Agreed. Anything that simplifies the build process and leads to less custom tooling is very welcome.

Great! This is my current build script:

BUILD_TIME=$(date +"%Y%m%d.%H%M%S")

CommitHash=$(git describe | cut -d- -f3)

GoVersion=$(go version | cut -c 14- | cut -d' ' -f1)

GitTag=$(git describe | cut -d- -f1)

FLAG="-X $TRG_PKG.BuildTime=$BUILD_TIME"

FLAG="$FLAG -X $TRG_PKG.CommitHash=$CommitHash"

FLAG="$FLAG -X $TRG_PKG.GoVersion=$GoVersion"

FLAG="$FLAG -X $TRG_PKG.GitTag=$GitTag"

go build -v -ldflags "$FLAG"

Please consider omitting build time from your artifacts. It makes build reproducibility a nightmare; people are coming up with frameworks to lie to build systems about the current time. https://reproducible-builds.org

Knowing where my build artifact came from and when is at least a good two orders of magnitude more important to me than my own binaries being reproducible.

Part of the reason why "reproducibility" has taken so long to appear is precisely that its use cases take a long time and a lot of scale to appear. I approve of distros pursuing this goal, but haven't got a lot of use of it for the stuff I build, which is why I have a very similar build script.

Please don't reply with a simple recitation of the putative benefits of reproducibility (I'm well aware) or some wild "what if" scenario relating to my own binaries that will occur once every few centuries on average. I have had multiple real problems in my life where not knowing the exact source checkout and time something was built has been a problem. I've only really once had a problem where I would have felt better if I could have reproduced something exactly, to be sure that the source and random unattributed binary were in fact the same, and even that ended up only as a "I would have felt better." (And now that I think about it, if said unattributed binary had just said what commit it was built out of that would have been fine too....) I am much more interested in solving the problems that actually manifest than problems that someone somewhere hypothesizes I may someday have.

Maybe we should have a specific .metadata section in assembly so we can ignore the timestamps when diffing for reproducible code.

If we allow that, attackers could hide a codepath in the binary that only activates for a specific build time in the .metadata section. That would defeat the purpose of reproducible builds.

That's an interesting scenario for sure, but "defeats the purpose" is a stretch.

Make it non-readable for the program.

> I have had multiple real problems in my life where not knowing the exact source checkout and time something was built has been a problem.

Once you know the exact source, what does it matter when the binary was built?

Additional context around the human environment when it was built; who was on the team? What was the problem trying to be solved? Does it match the source code commit dates I thought this was built from (i.e., if it was built a year after the commits, there's something going on there)?

I needed this in pathological cases in the first place; think "this stuff dates from when this mature company was just a startup" kinda things. Knowing when it was built wouldn't have changed anything about the executable itself but would have helped with "which strata of old employees should I be hitting up", what other bugs were present, whether we were using a broken compiler at the time, etc.

In a perfect world, sure, none of that would matter, but.... the sibling reply from mentat is right in principle, but in practice sometimes the pathological does happen.

However, if I had to choose between knowing the exact commit and knowing the exact time, I'd certainly choose exact commit, no question. But you don't; you might as well just stick all the data in.

> However, if I had to choose between knowing the exact commit and knowing the exact time, I'd certainly choose exact commit, no question. But you don't; you might as well just stick all the data in.

You don't have to choose between embedding the build time or commit information, but you do have to choose between embedding the build time or having reproducible builds. I know what I prefer, as a maintainer and consumer.

It's really nerve-wrecking (to me) when you pull down a binary from the website releases, pass it to sha256sum and gets something different than when you built the same commit from source. Usually that's because dates and times are embedded in the build, but sometimes stuff is just different depending on who built it, and that's scary.

I explicitly set the context as my binaries in my posts here. You aren't going to ever download my binaries from your linux distro.

For issues like these I have a tendency of just using xattrs.

To be fair, I'm aware that a poorly archived/copied file can lose them (especially over a network) and it's not part of `POSIX.1`. At the end of the day though I've found it's a reasonable "belt" to wear with your suspenders.

Changes in the build pipeline won’t be encoded in the git sha, since usually those are stored separately. And if you pull in a non-reproducible library, time will matter a lot.

This of course could be fixed with a Nix type system (or even Nix itself), where the inputs of everything are hashed. But that’s a much more radical change than what OP is doing.

> Changes in the build pipeline won’t be encoded in the git sha, since usually those are stored separately.

Build pipelines should be versioned as well. Correlate the application sha with the pipeline sha and you have both sources.

> And if you pull in a non-reproducible library, time will matter a lot.

I'd argue that if your application is pulling in a non-reproducible library and using it, then your application won't be reproducible until you've made the libraries you're using also reproducible. Otherwise the entire idea falls apart.

Agreed. It's like baking a cake and putting a best by date on it but having no idea if the ingredients you used we're expired or even rotten.

Git solves this problem by tracking all of the ingredients for you with plenty of metadata to ensure you're never baking a bad cake. Couple it with a changelog and tagging and you'll worry a lot less about bad or expired cakes.

When I see people depend on things like dates or build id's it's typically my first indication that something is wrong in the cake making process.

Yes, exactly. That OP needs to know the time is the problem being solved.

There are also cases where multiple VCS tools are used or where you might want to build from an archived full copy of source+vendored dependencies and might not even have an underlying VCS to speak of.

Reproducible releases should probably _not_ have these commit metadata built in (e.g. imagine reproducible building this 5 years in the future where you've moved off git). Though this is quite useful in debug builds where you might be have installed a number of different builds in the wild and want to be able to quickly figure out the exact version of the code a given binary was generated from.

I would include the commit metadata, otherwise you have no easy way to figure out where that source came from. The issue of out-of-VCS builds can be solved by putting the commit-hash into the directory/tarball name or a VERSION file, e.g. $PROJECTNAME-$(git describe --tags | sed "s/^v//").

When converting VCSs most converters will preserve the old hash in some form, e.g. by appending it to the commit message.

Github downloads however remain a problem, as they just include the branch name, not the commit hash.

Alternatively, just make it configurable:

  buildtime="${BUILD_TIME:-$( date -u +%FT%TZ%z )}"
  # Use $buildtime in your go build command.

This will only set the variable buildtime if BUILD_TIME isn't set already. Making reproducing a build easy: just define BUILD_TIME to be equal to the one of the binary you want to reproduce.

Please read the link. Reproducible Builds has standardized "SOURCE_DATE_EPOCH" for exactly this usecase. Introducing other variable names i just going to cause more problems.

https://reproducible-builds.org/docs/source-date-epoch/

It’s a proposed convention. A specification. But it’s not a standard.

Foxboron wrote "Reproducible Builds has standardized". Going to the website, they write "SOURCE_DATE_EPOCH is a standardised environment variable" and it links to https://reproducible-builds.org/specs/source-date-epoch/ which is a specification to standardize something.

So it's accurate to say that "Reproducible Builds has standardized" that since they only talk about their own organization, not that it has been accepted to IETF or something like that.

Man, if people just took the time to read the link and not the comment :)

I actually did and the link just references a specification, not a standard.

Something being “standardised” requires more than just a specification and just because the specification throws the term “standard” around it doesn’t mean it has been formally standardised.

This doesn’t mean people shouldn’t still follow the specification. Plenty of specifications exist that have not been standardised. For example, there is an RFC specification for CSVs but not formal standard.

So by all means push this specification as a de facto standard but please don’t mislead people saying it has been standardised when it has not.

It usually takes less than 30 seconds to copy+paste quote the portion of the linked content that's actually relevant. Doing so greatly improves the effectiveness of communication.

That’s a specification. Just because the specification says it’s standardised it doesn’t mean it is. Where’s the corresponding approval from ISO, ANSI or other standards authority?

There’s nothing wrong with a specification or even a reference implementation but they are not themselves standards. And that’s exactly what this is: a widely respected specification.

You shouldn’t have to lie about this being standardised for it still to be relevant.

You don't need to get approval from ISO, ANSI or any other entity in order to standarize something _within your own organization_. Again, the SOURCE_DATE_EPOCH variable has been standarized within the Reproducible Builds organization, and no one can say otherwise.

But that doesn’t still make the specification “standardised” for anyone else.

I hate arguing semantics because it’s usually a worthless argument. But the problem here is people are specifically arguing that Reproducible Builds have standardised something when they have not. If you still want to use their specification then that’s grand. You don’t need a specification standardised for it to be relevant.

However on the very specific point of whether this is a standard, I’m yet to see any proof that anyone has actually standardised the specification. All I’m reading is “but they say it is” and frankly that’s not enough to make a standard.

> But that doesn’t still make the specification “standardised” for anyone else.

Of course not, and no one has claimed that this is a globally recognized standard. The only thing that has been claimed is that they have standarized it for _themselves_.

It's important to get semantics right, and it's equally important to really read and understand what's being said, and not assume something "more" is being said than what is really said.

> Of course not, and no one has claimed that this is a globally recognized standard.

> > Reproducible Builds has standardized "SOURCE_DATE_EPOCH" for exactly this usecase.

> It's important to get semantics right, and it's equally important to really read and understand what's being said

I’d say either you’re wrong or the GP was unclear in his statement. Either way I don’t buy your attempt to shift the responsibility of error into me.

I wrote "Reproducible Builds has standardized". I made no assertions how this is recognized in the wider ecosystem.

However, a couple hundred projects heed this standard along with most distributions. Does your nitpicking really matter?

It does when you tersely react to other people’s comments stating they should follow “standards” (which you now finally agree are not a standard).

If your comment wasn’t so rude from the outset then I wouldn’t have nitpicked your statement. I think there’s some “law” people often quite about people who correct other people that applies to your post.

It's used by rpmbuild and dpkg-buildpackage (and The stuff underneath). That's pretty well standardized.

No, that makes it a common implementation or even a de facto standard. It does not make it standardised because Reproducible Builds is not a recognised standards authority.

There is no such thing as a "recognised standards authority". Who would be the root standards authority to appoint other standards authorities? ;) And why can't the Reproducible Builds project be one of them?

> There is no such thing as a "recognised standards authority".

Yes there is

> Who would be the root standards authority to appoint other standards authorities?

You’re thinking of signing authorities.

> And why can't the Reproducible Builds project be one of them?

You could also flip The argument on its head and say why should every committee that throws a specification up be considered a standards authority when half the time specifications are incomplete, contradict other specifications or flat out don’t work.

Anyway, I’ll make this my last post on the subject. I have no interest to waste any more time explaining the difference between a standard and a specification to people who are unwilling to consider there might be a difference.

> Yes there is

> You’re thinking of signing authorities.

No, I don't. I was asking for a list of authorities that you would consider authoritative (which invariably will be subjective, but would be interesting nonetheless).

> You could also flip The argument on its head and say why should every committee that throws a specification up be considered a standards authority when half the time specifications are incomplete, contradict other specifications or flat out don’t work.

Oh sure, specifications always suck™. But that is unfortunately quite independent from the organisation, happens all the time with ISO standards, IETF RFCs, or whatever you would consider "recognised" bodies.

> I have no interest to waste any more time explaining the difference between a standard and a specification to people who are unwilling to consider there might be a difference.

You haven't explained anything. All you have been saying in this thread is "I don't recognise the Reproducible Builds project as a standards authority, because I only believe in my personal list of authorities that I won't reveal".

The point is, a standard does not become a standard because it is published by a "competent authority". A standard becomes a standard when people in the respective field converge on using it.

And yes, there are of course certain fields where this convergence can be enforced e.g. through local law, professional associations etc. However, software development is generally not one of these fields: who will reprimand you if you write a C compiler that does not follow ISO/IEC 9899, or in this case one that does not respect SOURCE_DATE_EPOCH?

You just said "it's not a standard, it is a standard". De-facto vs formal body does not matter.

Why would you not follow their convention? That's how we've ended up with CLICOLOR and NO_COLOR. The NO_COLOR guy decided "nah it's not a standard". :/

You’ve missed the point. I’m not saying we shouldn’t have a specification not an agreed convention. I’m saying the GP shouldn’t get so argumentative about Reproducible Builds being a standard when it’s just a specification.

My message was more about how you don't necessarily need to exclude the date information as opposed to exact variable names. But thanks for the info.

no its not. build tools can use whatever names they want for environment variables. sharing tends to cause more issues then having namespaced ones due to unintended side effects.

I do that in all my Makefiles and it works great.

For timestamps I like to use something like,

  TZ=UTC git log -n1 --format='%cd' --date='format-local:%Y-%m-%dT%H:%M:%S'

There's probably a better way to do that, but the basic gist is to derive the most recent commit date.

Isn't this only a concern for OSS projects where reproducibility is a concern? At my proprietary coding job, reproducibility is a nonstarter anyway and build time is super important for debugging day-to-day (or even minute to minute) problems.

It's a concern for private industry as well.

The solarwinds and other recent supply chain attack involved injecting additional code into a build environment. Reproducible builds could have been done in multiple places as an auditing mechanism to catch this.

Another place where reproducible builds is useful in industry is being able to reproduce debug symbols for something after it has shipped. Sure you can rely on always saving debug symbols in an artifact repo, but if those age out and you get a customer case... what ya gonna do?

It seems that a proper reproducible build checker should omit this kind of metadata from the portions of an executable that it verifies to be the same.

It's hard to statically determine what parts of an executable affect the running of it. Many executable have timestamps in them that do affect how they run, so you can't just ignore all timestamps if you want reproducible builds. See https://reproducible-builds.org/docs/timestamps/

Proper does not mean an endless list of exceptions to deal with the endless number of ways to exclude effectively random, non-functionally relevant information in the binaries.

Because that where reproducible builds started.

Projects not interested in reproducibility can/should content themselves to live in the non-reproducible set instead of expecting someone else to monkey patch for them!

Will 2 binaries differing by timestamp hash the same? do you suggest that file hashing algorithms skip date bytes in a binary? The KISS solution is to not vary date in the binary.

Not sure why the timestamp of when the build happened should even be logged. Include the git repository, the sha and if on CI, the build number/hash and you're pretty much covered.

can nixos/nix package manager be somehow related?

Never include things that change on every build in the binaries. It is bad practice and you will regret it. You can let your CI/CD software keep track of it or write the info to a separated log file.

Only tangentially related, but go:embed is pretty neat https://blog.carlmjohnson.net/post/2021/how-to-use-go-embed/

Something I have been wondering is whether the Go bootstrap, wherein Go is itself built from Go 1.4, also needs to be updated so that Go bootstraps from Go 1.17 or 1.18.

The reason for this is the change in syntax for build conditionals. Go 1.17 being compatible with the old and new, but Go 1.18 will only be compatible with the new. By which point Go 1.18 will only use the new and Go 1.4 would no longer be able to build later versions of Go.

Versions after go1.4 up to 1.17 need to be built with 1.4. From 1.18 on, they need to be built with 1.16. See https://github.com/golang/go/issues/44505.

update: they missed implementing this and now go1.17 will be required to bootstrap go1.19+.

https://github.com/golang/go/issues/44505#issuecomment-94259...

iirc correctly their bootstrap always has been to bootstrap a Go 1.n version, that can build Go 1.n+1 etc.

Not always, there are gaps allowed.

I wonder what other languages/ecosystems embed the versioning into the binaries.

This reminds me of the mcs(1) command (Manipulate Comment Section) used for some Solaris 10 binaries to display the patch level information:

# /usr/ccs/bin/mcs -p /kernel/crypto/sparcv9/aes

/kernel/crypto/sparcv9/aes:

@(#)SunOS 5.10 Generic 150400-64 Dec 2018

This sounds useful, but this is a bad attitude:

> People who distribute binaries widely and build them doing funny things may now have more of those funny things be somewhat visible, though. Well, until they turn these features off with those documented flags.

Yeah only if they know to check! This is pretty much the unsafe-by-default anti-pattern. You know, mysql_real_escape(), yaml.load(disable_gaping_security_holes=True) and so on.

That said, the actual information looks fairly innocuous and file paths get accidentally embedded in C++ binaries all the time with no issues.

There used to be a lot more information in binaries in general. These days everything is stripped. Giving my age away here :)

Your comment indicates you didn't experience the era where binaries had nothing more than the machine instructions and data necessary to execute. If anything, I think binaries now are more bloated than ever.

I remember different. It used to be controversial when compilers of old had embedded some identifiers into the binaries they produced - like the infamous MSVC-added "Rich" header or I think Borland compilers used to have something like that too.

Some people used to strip or blank out all sort of identifiers for various reasons (IIRC, primarily to complicate analysis with automated tools).

Nice to see compiler support for something like that. We've been doing stuff like this in C(++) for decades, extractable with the `what` command. It's more fiddly without direct compiler support, but on the plus side you can adapt it to any VCS.

Can you access that auto-embed information at runtime with some stdlib calls?

They call it "uncommitted". What does that mean? Myself I have used "git describe --dirty" to store version information (not in Go binaries, but in general). "Dirty" will indicate changes to files under version control. But it will miss any new file that should be under version control but isn't.

Hmm... so many questions

  1. Go binaries are bloated already. Do they plan to compress this (and other textual info, like full type names) in the binary?

  2. I saw enough of build pipelines that do not clone repositories, but do a shallow copy for a tag, and have no .git folder. How much info will be added in this case?
  
  3. Moreover, pretty much every serious build has source generation steps (protobuf, jooq, etc - you name it). If resulting executable will have a very particular info about these, the sheer amount of information added will probably make executables considerable larger.

1. It seems to be very small amount of information, if you compare to what already being included in a typical Golang binary. What's 1kb (or even 10kb) when your binary size is already huge thanks to Golang? If binary size is very important for you, then you would surely use something else than Golang in the first place.

1) A lot of CI pipelines already do it by utilizing ldflags in go-build command. I don't think if few kilobytes more will matter much.

2): It will not be added (based on information provided in the article [0]).

3): Is it valid question? By my basic understanding of article the case of adding more bytes to binary will require having the main package [0]. Also, generation (e.g. gogo-proto, go:generate) will create only business logic code in new package, no additional metadata will be added.

[0] Quote from article:

> Version control information is embedded if the go command is invoked in a directory within a Git or Mercurial repository, and the main package and its containing main module are in the same repository.

OK, all I remember is that many people have complaints about Go executables being huge. How does this affect it?

Can you ever really trust this info? I think that the deployment manifest is the only thing you can really trust.

huh? how would a random file be more trust worthy than metadata inserted by the build tool itself at build time?

only way for this to be 'incorrect' would be if someone intentionally fucked it up or disabled it.

And if intentionally fucking it up is possible, can you really trust it?

If you are considering trust as a binary state then it's exactly as trustworthy as the manifest file.

If you consider as a continuum then it's strictly more trustworthy than the manifest file.

Trust to do what?

Not familiar enough with Go to know, would this aid hacking?

Reply on news.ycombinator.com