Go command support for embedded static assets (files)
go.googlesource.com
go.googlesource.com
The proposed feature is great, but the unwillingness of the Go team to use a separate, clearly defined project file or at the very least a separate syntax in your code file leads them to stuff every additional feature into comments, a space shared by human notetaking.
Let's have a look: * Build constraints (// +build linux) * Code generation (//go:generate <command> <arguments>) * Cgo, you can even stuff entire C programs in the comments (// #include <stdio.h>) * Cgo flags (// #cgo CFLAGS: -DPNG_DEBUG=1) * and now this, file embedding (//go:embed html/index.html)
Most novices would assume the commented out code does nothing, and rightly so in my opinion. Half of these features aren't even code-file specific but project-wide, making deciding which file to put them in hard and looking them up even harder.
I mean yeah, I think the comment thing is a bit wonky, and it's not the way I'd do it. But once you've got "//go:generate", adding "//go:embed" isn't really any worse.
EDIT: Just noticed this line in the proposal:
> Another explicit goal is to avoid a language change. To us, embedding static assets seems like a tooling issue, not a language issue. Avoiding a language change also means we avoid the need to update the many tools that process Go code, among them goimports, gopls, and staticcheck.
So this partly explains the idea of putting things in comments: things classified as "tooling issues" are put there so that "language tools" aren't affected. I do agree it probably would have been better to invent a specific language construct for, "This is a tooling issue, please ignore"; a bit like the # prefix in C.
There's way more than that: https://golang.org/src/cmd/compile/internal/gc/lex.go#L53
And I would argue that it's bull. Once they realised that they were introducing pragmas / attributes into the language, they should have bitten the bullet and actually introduced a clean way to annotate language items.
The first and second ones are excusable as "we'll only need one" and "well it's not worth the hassle" but at the third one it's not a special case it's a pattern. Especially as the excuse that "other implementations can ignore those" gets less and less true: a compiler which ignores go:embed can not be considered working.
https://www.reddit.com/r/golang/comments/hv96ny/qa_goembed_d...
These annotations are not go code but specific to this go source file.
The latter means that embedding them in the go source file makes them easy to find and read =, and avoid multiplying configuration files.
From there the former (adding annotations in comments) is basically the simpler, and perhaps only, way to embed the annotations while make sure the file is still valid go code.
Using comments (or “COMMENTS”) for these directives is sorta not ideal but if anyone could produce a case study where it causes anything but the most trivial of problems, I would be shocked. This seems like textbook overreaction.
@build linux
@generate <cmd> <args>
@C #include <stdio.h>
@cgo CFLAGS: -etc
@embed path/to/file
2. It allows the programmer to skip comments entirely, when reasoning about program behaviour
3. You can write comments without fear of the comment changing program behaviour (probably of little concern in practice, but still)
Putting directives in comments is always a hack. C and C++ get this right, with #pragma.
There's historical exceptions, but the new rule is this:
Comments begin with "// ", pragmas don't have that space.
Spot the bug in the below:
// +build windows
package my_pkg
If we had a normal pragma, "go build" could error out instead, but we don't.
Similarly, if you typo
//go:embd file
I think that's the main argument for not using comments.
#pragma C {
#include <stdio.h>
int foo(var * bar) {
...
}
// FIXME: This is kind of a hack, and we should
// get rid of it before v1.0
#pragma docs {
The FooBar function will Foo all the Bars passed to it.
}
func FooBar(...) {
Ah, well -- water under the bridge at this point. I like Go as a language overall. Most of their experiments have worked out pretty well, I think; it's inevitable that some of your experiments don't work out the way you expect, and then you're stuck with them.
There are plenty of purists that argue both for and against the concept but one cannot deny it fits very well in the sales pitch for Go. You can have a simple tool chain to ship a binary for any number of systems and that support for embedded assets would be a first class citizen.
I would also love to see them address more robust plugin options or officially adopt / endorse the pattern HashiCorp uses as they’re doing here with the bindata prior art.
You can pick it up in a week, sure. That's not the same as productivity.
Not unexpectedly, this means Rust is one of my favorite languages. Anything with a poor type system makes this approach unbearable, and even Python has better static typing than go.
In Python you don't know what you receive and someone could have modified something in an object somewhere in the code that will break down the line in unexpected ways.
Python does not have any type checking those are just annotation it does not enforce anything.
https://docs.python.org/3/library/typing.html
"Note The Python runtime does not enforce function and variable type annotations. They can be used by third party tools such as type checkers, IDEs, linters, etc."
If it's the former, I'd recommend https://tour.golang.org and The Go Programming Language.
If it's the latter... I can't personally recommend any beginner's textbooks which are written to use go, but it might be worth looking at Head First Go, the rest of the books in the series are quite approachable.
What would you say is the best starter language these days? Java? Python? Something else like C++ or Rust? I am not a coder at all. I mangle configurations and implement functionality upon request, but I always run up against scripts and CLI and I am no expert in bash or powershell and I don’t have to quit vim. But I get by okay. Not a dig, I just admit I only know as much as I do primarily from self-learning and doing. That’s why I care to learn what I’m not seeing because I’m not able to do it yet: the code.
There’s just so much to not know, especially about setups and small things that impart outsize benefits or functionality, like dotfiles for instance. All these things are so hard to learn in isolation. It’s hard to have scope and find the rails under you to know how to turn and move yourself around in the dev space, in a basic computing, nuts and bolts sense, if that makes any sense. So if you have any tips about things like that, let me know. For example:
They even link back, how nice!
When you feel comfortable with programming fundamentals in Python (everything from variables, functions, loops to classes and modules) I'd recommend looking into Go or Rust -- the latter is probably more of a challenge but it makes sense as a step up from Python.
My extremely personal opinions about the rest of the languages you mention, and these are my personal opinions so there's no need to tell me how wrong I am because personal opinions almost always are wrong (to someone else):
- Java: Highly sought after in the market but mostly found in legacy code bases, bloated (JVM) and not very well liked by developers apart from some scenarios where it's already used, or those that have been working with it for a decade.
- C++: Hated by everyone except those that are already very proficient in it and is quickly being surpassed by Rust, by no means a useless language but not one that makes sense to pick up from scratch in 2020 unless you have a specific reason to.
Would you say it’s worth looking into C++ due to legacy codebases?
If security is what you mainly care about though, maybe you'd just enjoy diving straight into rust rather than C++.
https://www.oreilly.com/library/view/ultimate-go-programming...
Go is a fairly horrible language by today's standards, but it does what it's supposed to. I like generating code automatically at compile time based on, but I can appreciate the appeal of something that forces you to actually do work rather than being clever.
Writing out source code by hand: Nothing beats TS or Kotlin. Not even Python.
But what surprised me with Go after a couple of month (and please, nobody should judge a language until having done a serious project): Taking everything into account, from installing to looking for libs, writing code, testing, compiling, revisiting code after a couple of month ... Go is an absolute dream in terms of overall efficiency.
Writing out error-checking or for-loops is a one shortcut in GoLand. Getting a table-test template for a function as well. People complaining about having to type this stuff out should learn a proper editor.
Exceptions are not less complex. They just move the complexity around. Exceptions as typically implemented also don't play well with event driven or concurrent models.
The thing I actually like about Go errors is that it makes you think about them right when they happen, while an exception encourages dealing with errors "later" which often means "never" or "as an afterthought."
Go kind of has exceptions, but panics are intended only for very extreme cases like out of memory errors that crash most applications and are often not recoverable. Using panics for non-fatal errors is bad Go code.
One reason I think people don't get Go is that it occupies a language niche that formerly was not occupied by any language. It's like Python meets C, a "low level scripting language." It's designed to be productive and pragmatic but fast and compiled and capable of dealing with pointers or even embedded ASM.
On codegen, I've never seen an IDE that will help everyone read what it generates. If we must generate code, do it during the build and then throw it away, don't make it something to maintain and review.
How fast you can churn out code in a language becomes completely irrelevant at those scales. How fast you can get to grips with a codebase, make the change, compile it, test it, and have it reviewed is much more important.
And that's where Go comes in; it doesn't have much cleverness so most code is instantly readable. Take a random file from https://github.com/golang/go/tree/master/src and any developer will understand it and be able to make changes. They optimized the language for compile speed (https://stackoverflow.com/a/8673468/204840); the joke goes that they came up with Go while waiting for a compile. And because the code is so simple and standardized, reviewing also takes less time.
I'm mostly thinking of Scala as the direct opposite of Scala which has as many different coding styles as it has developers, and Go's language design process as rebelling against most other languages that seem to want to put other language features in them for the sake of it (I'm still bitter about classes in Javascript and streams in Java).
TL;DR: If you're a solo developer or work in a small team, then Go may not be for you if you're measuring personal productivity. Its value starts to show if you scale up, and you go from churning out code to maintenance mode.
Rob Pike in 2015, about what Go was trying to avoid:
"Java, JavaScript (ECMAScript), Typescript, C#, C++, Hack (PHP), and more [...] actively borrow features from one another. They are converging into a single huge language." [0]
[0] https://www.dotconferences.com/2015/11/rob-pike-simplicity-i...
For me the reason is that the language imposes little cognitive load, leaving my brain free to spend almost 100% of its energy thinking about the problem I am attempting to solve.
With complex languages like C++ I find that I'm spending too much time thinking about language internals and syntax, and with C I am spending too much time doing super low level things. Both these languages also require constant attention to make sure their many built-in footguns do not go off.
Scripting languages like JS and Ruby can be more productive up front, but as a code base matures it becomes harder and harder to feel confident about the code not having hidden runtime bugs. Dynamic runtime typing is technical debt. These languages also run slower, though I have to say JS VMs are impressive.
I have not used Rust enough to comment. Feels like a better C++. Haskell feels like it would be great for certain areas where the provability it offers shines but isn't quite general purpose.
But YMMV. Ultimately I care more about what is written than the language. 'Tis a poor craftsman who blames his tools.
As a former Perl wrangler, I was super-productive in the short term with terse, elegant, and often clever Perl code: I could really express myself. Maintaining the codebase on the long-term was a nightmare though, so team productivity wasn't great. Go is the opposite, "harder" to write, but much easier to read and understand - which pushes up team productivity significantly.
Isn't that their unsubstantiated marketing term that keeps getting repeated.
If you squint it’s reminiscent of zeromq for IPC :)
You can contrast that with https://golang.org/pkg/plugin/
A tangential comment: is anyone else concerned a bit that discussions and debates happen on platforms like GitHub and reddit? These two platforms are large enough to be around for quite sometime, but are we making it easier to lose historical context because platform creators/designers are choosing third party platforms that they don’t host or control (or in some instances don’t or can’t pay for)?
This proposal allows "go build" to embed things in a very specific way, but it's not meant to be extensible.
Rust's 'include_bytes!' macro on the other hand is a macro in the stdlib that can be emulated in an external library. I'm fairly sure every feature of go's embed proposal could be implemented via a rust macro outside the stdlib.
For a specific example, I had a project where I wanted to serve the project's source code as a tarball (to comply with the AGPL license of the project). I was able to write a rust macro that made this as easy as effectively "SOURCE_TARBALL = include_repo!()" [0] to embed the output of 'git archive' in my binary.
Of course, there's a very conscious tradeoff being made here. In rust, "cargo build" allows arbitrary execution of code for any dependency (trivially via build.rs), while in go, "go build" is meant to be a safe operation with minimal extensibility, side effects, or slowdowns.
I've been working off and on on a language that tries to get the best of both worlds to some extent. The whole language is built around making sandboxing code natural and composable. Like Rust, it has a macro system, so lots of compile time logic is possible without adding complexity to the build system, but macros don't have access to anything but the AST you give them, so they are safe to execute. There's a built in embed keyword that works like Rust's include_bytes, which runs before macro expansion, which you can use to feed the contents external files to macros for processing. At some point I'll probably add a variant that lets you pass whole directory trees.
- go tooling (ides, etc) have to be taught about the _specific_ embedding in the same way one could teach rust tooling about specifically `include_bytes()` (or any other specific macro in the same way one teaches go tooling to handle specific pragmas)
In the world of rust build scripts, there is tooling that exposes information about which files are used if dependency info is all that is required (I don't know to what extent imperative macros are able to expose similar info).
The core of how I see the comparison here: if we restrict ourselves to the capabilities of go pragmas in rust, the same level of support is possible, but even without that restriction there are ways to obtain (though with more work) the same info.
So what you're saying is simplicity is only a virtue if it supports the thing you already liked?
> The path separator is a forward slash, even on Windows systems.
Which is much more ergonomic for developers than rusts treatment of os specific paths.
[1]: "Most" because they don't work for namespaced paths, by design. But note that the number of times you'll encounter those is limited, the Rust standard library doesn't handle them correctly in general, and a lot of other non-Rust software breaks when given them.
Something like: Pikes law of unfavourable comparison OR HN Golang oxidation rate
Go has a pretty high oxidation factor.
Or maybe they're just waiting for something to compile?
Not that it is or isn’t and clearly Go does things really well in some problem spaces but yeah, this seems to be a common theme for all young languages
I do think some members of the Rust community Take it waaay too personally, which I never saw in general when Go was the hip thing. That I agree can be very obnoxious
Java Resources - https://docs.oracle.com/javase/8/docs/technotes/guides/lang/...
Android Resources - https://developer.android.com/guide/topics/resources/providi...
.NET Resources - https://docs.microsoft.com/en-us/dotnet/framework/resources/
Win16 and Win32 resources - https://docs.microsoft.com/en-us/windows/win32/menurc/about-...
UWP resources - https://docs.microsoft.com/en-us/windows/uwp/app-resources/
C++ helpers for Win32 resources - https://docs.microsoft.com/en-us/cpp/windows/resource-files-...
Given that you mention bundles, Java's resource management shows the influence of Objective-C in Java's design.
Rust is from enthusiasts for enthusiasts. If the Rust community wants it to become more real / main stream, they need to look at how the Go team focuses on supporting devs for getting things done correctly with long term stability.
Generic compile time evaluation is an interesting feature, better than magic comments in my list. Though in this case it is compiler built-in [1].
[1] https://doc.rust-lang.org/src/core/macros/mod.rs.html#1122-1...
Not to pick on you or even the other child comments but the amount of people that complain about a language being compared to a modern equivalent is funny. What about every other thread on HN where the same thing happens? It’s alll turtles just enjoy the discussion
That said, I really don't like the further overloading of comments. Comments are for the human, not the compiler.
In go's case, it's manifest primarily through magic (magic files, directories, comments, names, env vars, switches, etc that you just have to know how to invoke). Another warning sign is the need for an external build tool like makefiles in order to build-in-one-command because you need extra steps beyond "go build" (such as "go generate").
IIRC Java eventually "solved" it by adding annotations as a language feature. I'm not saying Go should add annotations, but just sharing some language history.
And C / C++ has had compiler directives (e.g. #define) since forever, although there too (I believe) # is "just" a comment.
The fact that there are so many of these "embed file in binary" tools suggests that it really is a problem that could be usefully solved once, in a consistent and reliable fashion.
The only thing I'd want is to allow environment variables in the "go:embed" statements, while my assets might be in the same repo and thus relative, they may also be in a different asset repo (if I'm using git I could use git submodule, but if I'm using something else that might not be possible).
package assets // import "github.com/me/myproject/assets"
//go:embed *
var FS embed.Files
package mypackage
import "github.com/me/myproject/assets"
//use assets.FS
I was kind of hoping for:
package foo
//go:embed $ASSETS/*
var assets embed.Files
//go:embed * from $ASSETS
So, assuming your assets are in the repository github.com/me/myproject/assets, you could just do:
package foo
//go:embed */*
import "github.com/me/myproject/assets"
Then we don't even need a magic embed package at all. And generating a package would give us more flexibility, for example:
package foo // import "foo.io/foo"
//go:embed Name string "name.txt"
//go:embed Data []byte "bin.dat"
//go:embed "images" "templates" "html/index.html"
import embedded "foo.io/foo"
//use embedded.Name, with type string
//use embedded.Data, with type []byte
//use embedded.Files, with type fs.FSThe landscape of tools supposed to solve that problem isn't great. Things are changing way too fast with projects being archived or deprecated with mention to migrate to another solution which after another year is deprecated again. Because of those issue, I've went with go generate which isn't ideal but is at least stable. That proposal would be a game changer and hopefully it will become a reality as Brad is quite a prominent figure in the Golang world.
BTW. Fantastic software, filestash - really really good.
Instead of having a "messy" folder on their computer.
In short, these embedded assets are no more likely to be in memory than any other file-backed data. If you want to guarantee they are in memory, it is up to you to make that happen.
More important is avoiding having to answer the question "where are my assets".
If you don't need one of the features of having "moving parts" on the filesystem, then why would you want a dependency on the local filesystem?
You want your users to just be able to import that library and use it. The fact that you need a bunch of zone info files should be invisible to them.
Maybe those users are in a sandboxed environment that doesn't allow opening files, or you're in webassembly where files don't exist.
It's much simpler to just bake that data into the library.
Parent just chose timezones as an example.
This works with most editors and doesn't require fancy multi-language syntax highlighting support. When debugging complex scripts that involve many joins or CTEs having good highlighting can prevent a lot of headaches.
I can use this feature to bundle the report template, css, js, images file with the plugin to make download installation and use of the plugin much easier.
It also seems like an area where it'd be nice for there to be a single uniform way to do it.
For compile time, I make heavy use of xxd -i to generate header files in a range of projects. It is no longer installed by default on most +nix systems, but it is generally still in your package manager (often installed alongside vim), and has been around since 1990.
I resolved it, checked HN, and saw this was a peeve they wanted to solve. Hah.
I have a project I work on where I need to package a bunch of data that won’t be available locally. Given that I’m loading data, it seems most optimal to just package it as encoded go data from the get-go.
Right now I’m using go-bindata to embed gob files.
Go 15 is going to be a pleasure to use.....
I think this problem is best solved by a library.
Leave the damn language alone!
> One of the earliest and most popular was github.com/jteeuwen/go-bindata and its forks
There’s literally an entire section in the appendix dedicated to `go-bindata` and explaining what it generates. It was also first in the list of libraries mentioned.
I also disagree with you entirely. The document does a great job explaining the problems with having this “solved by a library”. I’m excited about this draft design.
This proposal does not change the language. "//go:" comments are comments because they are not part of the language.
If you’re considering bundling static assets into your binary for a service, you would almost certainly be better served by containerizing your service and copying those assets into the image.