Comment by Groxx | Hacker News Reader

Comment by Groxx | Hacker News Reader

>If the main module imports example.com/mod1/pkg1 and a separate example.com/mod1/pkg2 imports example.com/mod2, there is no way for example.com/mod2 to affect the build or run code on the developer’s machine, so you don’t need to consider it a dependency.

iirc (I do not have a test setup at the moment to verify) it does affect your dependency resolution, and therefore your build, though its code does not exist in your binary. I know this because those AWS / K8S / Google Cloud libraries cause MASSIVE problems with their constant breaking changes without major version changes, and their importing other libraries that also have frequent breaking changes without major version changes, even if the dependency those unused subpackages include (and therefore raise the minimum version) is only used by some other module that needs a lower version (iirc). It's quite a headache sometimes, and could be rather easily solved if you could set upper bounds and not just lower. Or if those giga-projects would stop doing such obviously bad things.

The version-affecting behavior is kinda unavoidable afaict. If they didn't include those unused version constraints, `go build ...` or just importing a new package within existing modules could cause your build to fail, forcing you to rerun version resolution. That'd probably just lead people to feel like "go is broken UGH", and go leans awfully hard towards avoiding that kind of thing. Mostly for the better, but not quite always / not in all ways.

It's tricky, to the point that I made a little playground to explore it.

https://github.com/FiloSottile/mostly-harmless/tree/main/dep...

The example.com/mod2 go.mod does not in fact affect version resolution, because it's not even fetched. However, it affects the example.com/mod1 go.mod, and the example.com/mod1 go.mod affects version resolution.

This doesn't help with the problem you are describing, but it still has value from a security point of view, because example.com/mod2 truly doesn't matter except to the extent that was already checked into example.com/mod1, which you do need to trust.

If you try to "go build" or "go test" something in example.com/mod2, you actually do get an error since Go 1.17, as if it was not in your dependency tree at all. You need to "go get" it like any new dependency.

These libraries are honestly so bad that whenever I have to interact with them I just split those dependencies into a separate binary :)

I just write my own, small, focused client for these AWS services since the SDKs are generally just so unergonomic. With vibe coding becoming a thing it's become even easier to do that.

You may already know these things, but for others who may not: the AWS SDK for Go suffers from shockingly bad discoverability.

For example, take the S3 library, github.com/aws/aws-sdk-go-v2/service/s3. If you have an s3.Client and you look at e.g. the ListObjectsV2 method, you might have no idea that there is a ListObjectsV2Paginator which makes it much easier to use, because nowhere in the method docs is it mentioned. Indeed, most operations that paginate have more ergonomic paginators, but none of them tell you this.

But that isn't even the worst of it. Say you want to download or upload a file to S3. If you haven't worked with AWS for other languages, you might think that you just do GetObject and PutObject. And yes, for small files, that's generally fine. But for large files you want to use resumable downloads and multipart uploads. So you look and lo, there is no simple way to do this in the AWS SDK for Go. But actually, there is! It's in a totally unrelated and unlinked package, called github.com/aws/aws-sdk-go-v2/feature/s3/manager.

Now you're getting some religion, so you ask "what are the other so-called 'feature' packages?" and you try to browse pkg.go.dev at the github.com/aws/aws-sdk-go-v2/feature level but nope, that's not a Go module so there's nothing to see there. In fact, you can't even browse the other s3 features, never mind find out what other services have features. Fortunately, you can browse their GitHub repo at least: https://github.com/aws/aws-sdk-go-v2/tree/main/feature

It's quite clear that they use poorly thought-out cross-language codegen for this, which partly explains the lack of ergonomics, but also shows that they don't much care whether you use their stuff properly.

Yeah the AWS PHP SDK is similarly bad. The Java SDK is okay but still completely lacking in examples for anything outside of the basics.

AWS seems to optimise for an SDK that be completely generated but not for an SDK that tells you what you want to know.

I dabbled in Kubernetes at work a few years ago. I was shocked at the state of their go.mod files. Especially because I had heard Kubernetes used as an example of great Go software. Right off the bat I needed to copy a dozen or so `replace` statements into my go.mod. I was sure I just didn’t understand something and spent an hour or so looking for the “right way” only to discover other open source controllers that had the same replaces in their go.mods.

Maybe that was a transitory stage and it’s straightened out now. I certainly hope so.

Definitely do not look at Kubernetes as a good example of go code and especially not how it handles deps (btw, it only relatively recently switched to go mods).

Not to say it is a bad project. Not at all.

I think a lot of people assume Kubernetes must be a good Go example, because it's so big and successful. But it started life as a Java project and was ported into Go, and that shows in some of the architecture, Also, in general across all languages, picking the absolute largest projects you can is often not a great idea in terms of copying design unless you too are going to be that large, e.g., I wouldn't suggest using Firefox as an example of C++ necessarily. Such projects always end up developing solutions to problems you will never have, and solving problems we don't have is one of the most common mistakes software developers make.

I like your comparison about Firefox not being a great example of "good" C/C++.

I'd personally consider myself comfortable in C/C++. I've built Wayland compositors, H264 backends for live-streaming, and built Chromium occasionally for testing. Despite being a die-hard Firefox (zen) user - I still have not been able to compile Firefox! To be fair, this was pre-Firefox quantum days so I hope their SVM and build tools have improved.

I remember early on the Go people complaining about the K8S Go code

They write it in space shuttle mode. Which means that it’s all got too many components from too many makers all kludged together and then one piece falls off and the whole thing blows up in production.

godep and vendoring completely solved this problem.

Go modules relies too heavily on dependencies being good citizens, which is a very naive approach to dependency management.

You can still do vendoring with Go modules, using the `go mod vendor` command.

I'm a bit out of date now, but 'go mod vendor' did not in fact vendor all dependencies when I last used it. It seemed more like a CI caching mechanism then actual complete dependency vendoring.

This is not the experience that I have. I use vendoring heavily because it's nice to be able to review all the relevant code changes when Renovatebot updates a dependency [1], and to get a feel for how heavy or lightweight certain dependencies are. If vendoring was incomplete, I would see it trying to download missing dependencies at compile time, but the "go: downloading $MODULE $VERSION" lines only show up when I expect them too, e.g. during "go get -u" or "go mod tidy/vendor".

[1] Before you ask, I'm not reading the full diff on something like x/sys. Mostly on third-party dependencies where I find it harder to judge the reliability of the maintainers.

I'll throw in a +1, afaik vendoring is a complete and reliable solution here.

Go's `mod`-related commands have had quite a large number of breaking changes in behavior through the years, so I won't say it's a stable solution (aside from completely disabling modules, that has always worked fine from my use). I've had to fix build scripts at least once or twice a year due to that, e.g. when they started validating that the vendor folder was unmodified (a very reasonable thing to do, but still breaking). But once that is overcome it has always worked fine.

Reply on news.ycombinator.com