How to start a Go project in 2023
boyter.org
boyter.org
go mod init mymodule
- Vendoring dependencies using "go mod vendor" is not a good default workflow - it bloats the repo, the checked in code is impossible to review, and is generally a pain to keep up to date. Don't, unless you really have to.
- There's no point in stripping a binary or even using UPX on it unless you're targeting extremely low memory environments (in which case Go is the wrong tool anyways), all it'll do it make it harder to debug.
Worse, if you run multiple instances of the same binary, none of them can be shared.
A bit simplified, without UPX, 100 processes of 100 MB binaries requires only 100 MB RAM for the code, but with UPX 10 GB.
Edit: In reality, likely only a fraction of that 100 MB needs actually to be mapped into memory, so without UPX true memory consumption is even less than 100 MB.
If the application is uncompressed, the uncompressed executable will be loaded into ram 1 time and be reused by every instance of the application.
And even then, it's of dubious value when game install footprints are overwhelmingly dominated by assets rather than executable code.
However, when a binary is compressed, this cannot work, because in the file the binary is represented as a compressed data. The only way you can work around is to allocate some memory, decompress the binary there, map the region as executable and run it from there. This results a non-shareable copy of the data for each running instance.
A random link about this issue in practice: https://github.com/nushell/nushell/issues/4131
Deploing at a large enough scale, perhaps where other optimization options aren't as good or even available, could also be a target.
unfortunately most teams don't schedule a periodic `go mod tidy` so you just end up with ancient deps
most people never read the code of the deps they pull in, so I don't think vendoring provides any security assurances
If the package has bugs, you're far better off either waiting for upstream fixes, working around the bug in your application code, or just switching to a different library. That goes double if the library you're using is missing a feature you need, even if it's scheduled for the next version release.
Unless you're prepared to maintain a full-on fork of the dependency (and, if you do, please make it public), everything about vendoring for these reasons is 100% bad for you for very little incremental benefit. It's like the joke about regular expressions ("You have a problem and think 'I'll use regexes to solve it.' Now you have two problems"), except it's not a joke, and it sucks way more.
TL;DR: Vendoring to cache for CI/build servers, yes. Any other reason, just don't; it's not worth the headaches.
go.mod/sum files already remove that confusion as it’s their intended purpose
I really dislike absolutes like this.
My target is 30,000+ servers and distributing a binary to all of them is a lot easier when it is 3m than when it is 26m.
The updates are not pushed, they are pulled. Why? Because the machines might be in some sort of rebooting state at any point. So trying to first communicate with the machine and timeouts from that, would just screw everything up.
So, the machines check for an update on a somewhat random schedule and then update if they need to. This means that a lot of them updating at the same time would also saturate the network.
Smaller binaries matter.
What’s your use case for having machines on 100Mb? Are you using GbE hardware but dropping down to 100Mb, and if not, where are you getting the hardware from?
Sounds like you might work in a really interesting domain :)
Which can also be solved with compression at various other stages of the pipeline as mentioned by other commenters, but just to say that that's an easy case where this matters.
They were for mining ETH... we've turned them off though now that PoS has been successful.
IBM used to use a variant of Bittorrent to internally distribute OS images between machines. That was more than a decade ago though, when I was last working with that stuff.
Another issue with that is that the systems I was running can go offline at any time. P2P, which could work, kind of wants a lot more uptime than what we had. It would just add some complexity to deal with individual downtime.
machine <-> cloudflare <-> github
CI would run, build a binary that was stored as an asset in github. Since the project is private, I had to build a proxy in front of it to pass the auth token, so I used CF workers. GH also has limitations on number of downloads, so CF also worked as a proxy to reduce the connections to GH.
I then had another private repo with a json file in it where I could specify CIDR ranges and version numbers. It also went through a similar CF worker path.
Machines regularly/randomly hit a CF worker with their current version and ip address. The worker would grab the json file and then if a new version was needed, in the same response, return the binary (or return a 304 not modified). The binary would download, copy itself into position and then quit. The OS would restart it a minute later.
It worked exceptionally well. With CIDR based ranges, I could release a new version and only update a single machine or every machine. It made testing really easy. The initial install process was just a single line bash/curl to request to get the latest version of the app.
I also had another 'ping' endpoint, where I could send commands to the machine that would be executed by my golang app (running as root). The machine would ping, and the pong response would be some json that I could use to do anything on the machine. I had a postgres database running in GCP and used GPC functions. I stored machine metrics and other individual worker data in there that just needed to be updated every ping. So, I could just update column and the machine would eventually ping, grab the command out of the column and then erase it. It was all eventually consistent and idempotent.
At ~30k workers, we had about 60 requests per second 24/7 and cost us at most about $300 a month total. It worked flawlessly. If anything on the backend went down, the machines would just keep doing their thing.
Always go with the simplest solution first.
3mb is after `xz -z -9e`.
But, if you start with something smaller, you generally get something even smaller.
I tried UPX, but ended up with just `-s -w` (and xz), simply because UPX was taking too long to build the binary in CI.
More importantly though, I was responding to OP's absolute.
Vendoring would make it more likely you're gonna review the changes, be ause you can quickly eyeball whether or not changes look significant, which is something you often won't get out of a go.sum change.
There's a generic question of how you build confidence in your dependcies not being compromised, and there's steps you can take to mitigate that without reading code, but if everyone was adopting that stance then we'd likely have no mitigations
Now you can set up a proxy server; however, I don't want to do that. I'm pretty sure I have a few vendored packages that no longer exist at their original import path. For code reviews, we put off checking in the vendor path til the end if possible.
Always vendor your dependencies in your private Git repo or a proxy you control. Or heck, even in some long term backup solution if you must. Experience trumps theory.
Golang now has an automatic transparent caching proxy at pkg.go.dev. If your build has ever worked, it should continue to work even if the original source goes away. Furthermore, your build should only break if both pkg.go.dev goes down, and the upstream source is unavailable (is down or has moved).
This may require a few extra tricks to plumb through, for example, to make all cmd's be externally importable (i.e. in the project repository, transform "cmd/foo/%.go" from being an unimportable "package main" into an importable "cmd/foo/cmdfoo/%.go", then have a parallel "cmd/foo/main.go" in the vendoring repository that is just "func main() { cmdfoo.Main() }", same as you have in the project repository in fact).
Vendoring aside, this is also a useful pattern if you're "go:embed"ing a collection of build artefacts coming from another source, like a frontend HTML/JS/CSS project.
This lifecycle is vastly cleaner and easier to update/control than vendoring, and also forces you to actually have explicit copies of everything your build needs in the same way that vendoring does, but in a cleaner, separated, traceable, manageable way.
Go's setup is that if you don't vendor your dependencies then your build might break at any time, no?
> Why did a previously available module become unavailable in the mirror?
>
> proxy.golang.org does not save all modules forever. There are a number of reasons for this…
(I am a googler, but don't work on the go team – my opinions that projects should vendor and actually review their dependencies are my own)
If you're vendoring something without an appropriate license, you're skating on thin ice legally.
Unless you're doing something stupid like "create a clean virtual environment for every build" then yea your build might break if you lose the internet or the packages disappear. Just don't ever do that stupid thing.
You're not expected to review the committed dependencies any more than you're expected to review the external repositories every time you update go.mod/sum. If you don't care, just ignore those parts - if you do care, you were already doing it.
Vendoring dependencies is a nice way of using private Go repositories as dependencies in CI builds without importing any security keys. Vendor everything from dev machine, and build it in CI. You don't even need an internet connection.
While there are valid arguments against vendoring dependencies, I’m not convinced this is one of them in the typical case. It’s exceptionally easy to ignore certain directories when reviewing PRs in GitHub (although I still wish this was available as a repo-level preference), and I’d hope at least this would be the same in Gitlab, BitBucket, etc. I don’t review vendored dependencies, and I wouldn’t expect anyone else to, although the utility of that is admittedly domain-dependent.
Go also has the benefit that its dependencies tend not to be in deep chains, so the level of repo bloat when vendoring is usually not too terrible, at least relatively speaking.
But WTF is this about not reading your dependencies. Read your dependencies! It is the most amazing superpower for someone to be like “Uh I don't know how Redux handles that and you can just tell them because you have read Redux. And that's also how you'll know, hey, do they have tests, are they doing weird things with monkeypatching classes or binaries at runtime, “oh the request is lazy—it doesn't get sent unless you attach a listener to hear the response,” what would it look like for the debugger to step through their code and is that reasonable for me to do or will I end up 50 layers deep into the call stack before the code actually does the thing.
I get it, this dependency is 100,000 LOC and if you printed it out that's basically 5 textbooks of code, you'd need a year to read all of that and truly understand it... Well don't use that dependency! “But I need it for dependency injection...” I mean if that's all then use a lightweight one or roll a quick solution in a day or explicitly inject your dependencies in 5 pages or or or. My point is just that you have so many options. If that thing is coming at 5 or 50 textbooks or whatever it is, what it actually means is that you are pulling in something with a huge number of bells and whistles and you plan on using 0.1% of them.
That is, when your code is compiled, the linker can prune code that is never called. Then a feedback mechanism could show which part of the code is actually used (like looking in the .map of the linker).
Does something like that exist?
- https://github.com/golangci/golangci-lint - meta-linter
- https://goreleaser.com - automate release workflows
- https://magefile.org - build tool that can version your tools
- https://godoc.org/github.com/ory/dockertest/v3 - run containers for e2e testing
- https://github.com/ecordell/optgen - generate functional options
- https://golang.org/x/tools/cmd/stringer - generate String()
- https://mvdan.cc/gofumpt - stricter gofmt
- https://github.com/stretchr/testify - test assertion library
- https://github.com/rs/zerolog - logging
- https://github.com/spf13/cobra - CLI framework
FWIW, I just lifted all the tools we use for https://github.com/authzed/spicedb
We've also written some custom linters that might be useful for other folks: https://github.com/authzed/spicedb/tree/main/tools/analyzers
- https://github.com/charmbracelet/wish - Golang SSH server that makes building SSH apps easy
- https://github.com/charmbracelet/vhs - terminal GIF demos
- https://github.com/charmbracelet/log - minimal and colorful go logs
- https://github.com/charmbracelet/gum - leverage charm's bubbles to write useful TUI shell scripts in any language
Might be a little out of date.
https://github.com/shimman-dev/piscator
What did you find complicated about it? What I struggle with is creating man pages automatically, although I did just find where in the repo this is explained.
subcommands does look neat tho, I'll likely use it for another tool I have in mind.
One thing I did not realize is that cobra and charmbracelet/bubbletea are not compatible. It may be my inexperience in making CLTs vs TUIs but I was disappointed I could use say the loading spinner from bubbles easily in my tool (opted for briandowns/spinner instead).
I like Cobra because it gives you a great place to start, with tons of stuff "for free". Things like spellcheck on commands like if you type "mycli statr", you might get a response "mycli statr command not found. Did you mean mycli start?". This is out of the box. I don't do a single thing to create this functionality. Really nice help pages, automatic --help flags with autodocumentation all comes for free, by just running a simple init script to start the project. It speeds up my ability to make a reliable CLI because I don't need to worry about alot of the little things, I basically just create the command I want, choose the flags, and then start writing the actual code that performs the task and don't have to write very much code to manage the tool.
I usually organize my project so all the Cobra stuff is in the main module. Then I write my own custom module that contains the application code that I am building. It builds great seperation between the two. The main module just has cobra setup, flags, configuration, documentation, etc.. Then for each command, all I do is call a function from my module, where all the logic resides.
This makes it easy for me to switch between a "Cobra" context and my "Application" context depending on the module. It also makes it portable. If i want to use a different CLI framework or make this into a backend job, I can pull my module into a different project and then call the functions from that other project that reside in my module. The module is unaware of Cobra entirely, but it performs all my logic. The cobra module (the main module) contains only Cobra stuff and then offloads all the logic to my application module.
Cobra has all the power you could want from even the most advanced CLIs (I think github's CLI and Kubectl, kubernetes cli are both built on it for example). But you don't need to use any of the advanced stuff if you don't want. It means there is a lot of confidence to build a project in cobra and if it grows you won't be limited, but it also abstracts the complexity away when the project is simple.
I don't have a dog in this fight, just a fan. It is a tool I really appreciate. I will check out subcommands though, it looks like a good project. Reminds me of "click" for python.
I prefer to just type "fuck":
For this there is also the testcontainers project, which has support for Go and other languages.
Great for integration testing.
how should you actually start hiking? grab a water bottle, get outside, and hike.
here is how you should _actually_ start a go project in 2023:
$EDITOR main.go
go run .
Then maybe write something on the specific topic in detail than bundling them into a How-To tutorial. A How-To tutorial is suppose to show "how to" do something correctly, it's a teach than showcase.
I do understand that the author was writing this article with best intentions in their mind, but the resulting article is not a How-To, rather, it's a How-Do-I which is opinionated.
I think when reading articles like this, it is important to remember that, sometimes, more is less.
You import this many tools into your project, many of them are unnecessary and will not help you completing the project, now they've been download and installed, maybe they're even interfering with other tools. You look at them and starts to think "hey, make be I should learn to integrate and utilize them". Then you wasted an afternoon trying to utilize the tool, but by the evening you realized that in order to use the tool correctly, you must restructure your project.
This is not how you can finish things, you know? If you want to write a new project, just `go mod init` it and write the code. And during the writing, if you found the need for some tools, just introduce those tools one by one to fulfill the need. Don't downloading tools or creating "project layouts" just because some tutorial said so.
go mod init <project name>
go run main.go
Along with the issues listed here you will run into issues with editors not building/linting your tests files because they have build tags that the editor is unaware of.
You can also put the environment variable in a TestMain[1] to cover an entire package of integration tests:
func TestMain(m *testing.M) {
if os.Getenv("RUN_INTEGRATION_TESTS") == "" {
fmt.Println("Skipping integration tests")
return
}
os.Exit(m.Run())
}
That way you can blend unit and integration in the same package.
That's completely fair; it's still in development so this isn't a complaint! But just saying, at this point you need to be prepared to have to deal with that.
There was a last round of changes mostly revisiting use of contexts a few months ago - hats off to jba for taking a lot of time to work out the best fit
and here is a link to the draft release notes https://tip.golang.org/doc/go1.21
RUN CGO_ENABLED=0 go build -o mybin -ldflags '-extldflags "-static"' -tags timetzdata
FROM scratch
COPY --from=app-builder mybin mybin
ENTRYPOINT ["/mybin"]
COPY --from=alpine:latest /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/[0]https://github.com/mitranim/gow [1]https://github.com/cespare/reflex
# Makefile test: `find * -name "*.go" | entr bash -c "clear; go test ./..."`
one minor thing: I've skipped using build tags for integration tests because those tests will be out of sync one day with your main code, even with Goland (?).
Instead I use the usual test and check if an environment variable is set, if not, then
t.Skipf("env var %q not set, skipping integration test",envVarName)
func init() { flagIntegration := flag.Bool("test.integration",false,"run int tests") }
$ go test -v -test.integrationWhat do you mean here? What would be out of sync, and what would happen if it were?
I used to use buildflags before this, but my linter ignored those files so they were hard to maintain
# show the impact of cutting any package
goda cut ./...:all
goda has a bunch of other capabilities around dependency analysis, and was written by long-time Go contributor Egon Elbre. The examples in the README are the best way to get started after 'go install github.com/loov/goda@latest'.
Minor improvement over "mkdir -p $GOPATH/src/path/in/vcs/remote/<stare at the terminal endlessly trying to decide on a name>"
/s
(Go comes batteries included)
We ended up moving the project to a DI framework with an ORM-ish library since things got out of hand.
"It's a special purpose hook that is almost never needed".
[0] https://groups.google.com/g/golang-nuts/c/qDhJbkE1QeY/m/JoV2...
GOBIN := $(shell go env GOBIN)
And then use that var where needed.
mkdir new-repo
vim new-repo/main.go
Between that, goreleaser, 2-stage dockerfiles, static binaries, etc. It all just works so well and only needs Go for most things.
Recently for actual stuff we have been using Exho, Zerolog, and fairly common libraries for most tools.
And then for GUID generation we use https://github.com/oklog/ulid instead of https://github.com/google/uuid/
But it overall seems much nicer than running node/js or python on the server side, no?
There are a lot of foot guns. nil slice? Fine. nil map? Segfault. Loop variables with closures. The list goes on.
Generics seemingly split the community. May be some libraries won’t get used because they picked the wrong side.
It’s surprisingly weak at modeling data. Union types would really help out.
The community is so anti-design that it’s hard to play with them. Most want to make a big ball of mud and call it agile. When you point out simple patterns, they call you an Architect Astronaut. Checkout r/golang. Also look out for people telling you how dumb you are for wanting generics.
In many cases it’s a step backwards but it has the positives you posted. That is often a reason to grin and bear it. Eventually Stockholm Syndrome kicks in.
indeed a brutal footgun but will be fixed soon
A fix for the loop variable closure problem is now in the official proposal process, which is how language changes happen in Go.
It’s a concrete proposal from the core Go team and seems to be on track for acceptance:
https://github.com/golang/go/issues/60078
An implementation is already available on tip and in the upcoming Go 1.21 release behind a GOEXPERIMENT flag.
The community reaction has been extremely positive. As one approximate measure, an earlier draft of the proposal had 671 upvotes and with 0 downvotes:
I haven't really observed that at all.
One thing that is going on is there hasn't been a massive disruption while everyone stops to rewrite the world in generics, and generics are not suddenly everywhere, which is what some people had predicted would happen. I think part of the reason is that in some cases another solution (closures or interfaces or whatever) can be a better fit, and the evolutionary approach to generics that Go took means you can use generics in conjunction with non-generic libraries or other pre-existing approaches without suffering from an ecosystem split.
there is nothing wrong with Go; it delivers on its promise, you don't need to be a genius to use it, has good community support, and you can get access to a large and decent job market
Rust is a great tool but isn't as purpose-suited to network services as Go
Zig is even less purpose-suited to writing network services and won't be at Go's level of maturity for years, if ever
If a backend dev could only know one language in 2023, it would be hard to go wrong with Go
Why?
I’m not putting down any other development environments, but everything you say is absolutely true in your circumstances (mine too).
Using Go as a PHP alternative is pretty much the use case most aligned with its niche. So go nuts if you like doing that. But stray too far from that use case and Go will start to provide pain without adequate justification, especially when compared against Zig, Rust, or even TypeScript.
> optimized for junior programmers to write babby's first enterprise
This is the (toxic) attitude Go strives to distance itself from. There is no magic, we can all be equals in this place. It's humbling. I'm not aware of any other mainstream project that captures this essence so well.
There is power in a language equalizing things. If you aspire to wring elegance out of complex or esoteric language features then by all means have fun with that but I have no interest in working with you on that. Your definition of pain could not possibly be more diametrically opposed to mine.
...in the Harrison Bergeron sense.
The fact that Rust has attracted relatively inexperienced coders to do bare-metal, real-time programming shows that you don't need to nerf the language in order to appeal to interested developers of all skill levels.
It's the junior engineers that most often struggle with trying to devise a way to use every language feature under the sun when solving a problem, not the other way around.
No, it's just trade-offs.
I think you are making the same mistake by looking for validation on HN that you're making some sort of Better Choice, but you're just making a normal choice. You just don't yet have the experience to see all the trade-offs nor how they compare to, say, Node or Python.
For example, there are various ways Node is "nicer" than Go on the server. Just compare things like Promise.all or a concurrency-limited Promise.map to Go's WaitGroups.
I think it appeals to cynical devs who have seen projects misuse more powerful languages, and who don't want to debate style guidelines or linter settings for any more than 5 minutes. I count myself among them.
I personally hate the empty interface and definition shadowing of Go but that could be just me not “getting it”. Fortunately at work we don’t use that too often
I think most of the criticism is from people like me coming from C++. I am continually baffled that people write web backends in Python and Node at all, to me they seem so inappropriate that criticizing them would be a waste of time. I would consider Go to be much much better overall, and thus worthy of actual criticism
Care to elaborate? I'm curious what's wrong with either for web backends
There have been about a hundred “why Go”/“why not Go” threads on Hacker News already.
Which languages did Go beat out?
Nowadays, it is trailing most languages in expected set of features.
Go isn't the only alternative to running node/js or python on the server side.
Would there something similar for a next, second, step that focuses on go concurrency aspects?
Go (and erlang/elixir) as touted as concurrency-ready platforms and good overviews of setups, tools and best practices might help more people benefit.
In my own experience coming from a Java background, I find Go much easier to build from scratch with since the control flow is so plain and the standard library API is simple and well designed - worth trying.
I personally have a habit of updating only when the next patch version is out. It saved be countless hours of debugging and frustration.
2. read&write the f*k source code.
nothing else
Really thats all you need to get started. OK: a text editor, the Golang toolchain and ideally a terminal/shell.
Worry about other things if/when they come up, right in your face. Assume by default: YAGNI.