Indie game Haunts unable to compile its Go code
forums.thedailywtf.com
forums.thedailywtf.com
Submodules are flawed in a lot of ways, but as a simple way to to keep a pointer to the appropriate version of an external project they are great. It's a little extra work to set them up, but it's fairly marginal. Check them out with go get, add the repo after the fact with git submodule, and after that just use git submodule to always grab the correct versions.
1. A tag named for the version of go used, or 2. A branch named for the version of go used, or 3. The master branch
But no, you can't spec a version to import explicitly.
This is a decided on feature by the golang team, not an oversite or something beyond their technical capabilities.
Go badly needs two things:
1. A best practice that dictates that you never import from remote repositories in production, long-term code; the feature is fine for one-offs and experimentation, but the article summarizes only one way this style of work can lead you in to a maintenance world of pain. What happens if the repo you're importing from Github is deleted? What do you do for fresh clones? You're going to end up changing the URL anyway. I feel the Go community has kind of glossed over this (and I like Go).
2. An equivalent of CPAN or PyPI, which you could then import from in concert with a tool to manage those dependencies, a la:
import (
"cgan/video/graphics/opengl"
)
Developer A checks out the code clean. Five minutes later, developer B checks out the code clean. In both cases, your "go get" bootstrap script fetches two different commits, because in that five minutes, upstream committed a bug. Developer B cannot build or, worse, can build but has several tests fail for unknown reasons or, even worse, the program no longer functions properly. Developer A has none of those problems. In a world with a CPAN-like, developer B can see that he has 0.9.1 and developer A has 0.9.0, developer B can commit "foo/bar: =0.9.0" to the project's dependency file, then everybody else doesn't suffer the same fate. In the current world, you're either massaging your local fork to keep the new commit out, or any other troublesome, non-scalable approach to this.
Building large software projects against a repository never works. You need tested, versioned, cut releases to build against, not master HEAD. It only takes one bad upstream commit to entirely torpedo your build, and you've now completely removed the ability to qualify a new library version against the rest of your code base. Other people are suggesting "well, maintain your own forks," so you're basically moving merge hell from one place to another. I, personally, have better things to do with my time; I've seen (Java) apps with dozens of dependencies before, and keeping dozens of repositories remotely stable for a team of people will rapidly turn into multiple full-time jobs. Do you want to hire two maintenance monkeys[0] to constantly keep your build green by massaging upstream repositories, or do you want to hire two feature developers? Exactly.
I've started writing a CPAN-like for Go a couple times but I'm always held back by these threads:
https://groups.google.com/forum/?fromgroups=#!topic/golang-n...
https://groups.google.com/forum/?fromgroups=#!topic/golang-n...
The second one highlighting how difficult Go is to package as a language -- my personal opinion is treat Go just like C and distribute binary libraries in libpackage, then the source in libpackage-src. If one message in that thread is true and binaries refuse to compile with different version compilers, I'm troubled about Go long-term.
[0]: I'm not calling all build engineers maintenance monkeys. I'm saying the hypothetical job we just created is a monkey job. I love you, build engineers, you keep me green.
And in cases where that's not obviously the case, my habit is just to fork the repo and "go get" my own fork. I can then update it as needed.
Not saying these are optimal solutions, but they are solutions. I do personally wish I could "go get" a specific commit or tag.
Your second point I am less sold on. Transitive dependencies (pkgA imports pkgB@v1 but my code need pkgB@v2 which is incompatible with v1) are the thing of nightmares in large systems development, which is what Go is designed for... that lack of versioned imports wasn't an oversight, it is a feature.
Centralized repos are centralized points of failure, and only as good as they are well managed. NPM versus CPAN if you will. Any serious project will localize dependencies, even if they are in CPAN, you never know when CPAN will be down or other unforeseen things might happen.
And if you understand the basics of probability, that's not a good scenario.
[1] http://search.cpan.org/~dagolden/CPAN-1.94_65/lib/CPAN.pm#Cr...
The basic idea is to commit everything under $GOPATH/src so that you never need to touch the net to reproduce your build. After running "go install" you should check in the third-party code locally, just like any other commit.
Then updating a new third-party library to sync with their trunk is like any other commit: run "go install", test your app, and then commit locally if it works. If it doesn't work, don't commit that version; either wait for them to fix it, sync to the last known good version, or patch it yourself.
If you aren't committing your dependencies then you're doing it wrong.
Fortunately there are alternatives (go getting a personal fork, etc.) that aren't entirely anathema to maintainable long-term development.
Disagree strongly (and hate absolutes like "doing it wrong"). Their metadata, yes, by all means, commit that. There is absolutely no reason, however, to have the source code for a dependency in my tree, Go or not.
Give me a binary I can link against, or at least the temporary source in a .gitignored location, and let's call it a day. When I want to bump to a new version my commit should be a one-line version bump in a metadata file, not the entirety of the upstream changes as a commit. I've seen a sub-1MLoC project take 10 minutes just to clone. Internally! You're telling me you want to add all the LoC and flattened change history of your dependencies in your repo? Egads, no thanks! Where do you draw that line? Do you commit glibc?
There's just no reason to store that history unless you are in the business of actively debugging your dependencies and fixing the problems yourself, rather than identifying the issue and rolling back to a previous version after reporting the problem upstream. I guess it's paying your engineers to fix libopenal versus paying your engineers to work on your product; one's a broken shop, the other isn't. Some people will feel it's one, some the other.
It makes me weep when I hear gophers telling me to vendor all my dependencies. That is insane. Completely insane.
- incompatibilities that were introduced in the version 1.2.1 while you've only tested your code with 1.2
- the host is down so you can't compile your own code because your dependency is not available
- the host is hacked and "foo" was replaced with "malicious foo"
- exponential increase of testing (you really should test with all version of your dependencies you use)
Ultimately, I don't understand the doom and gloom point of view. C, C++, Java, C# etc. programmers have been pulling dependencies in their repos for ages. In my SumatraPDF I have 12 dependencies. I certainly prefer to manually update them from time to time than to have builds that work on my machine but fail for other people or many other problems that are a result of blindly pulling third party code.
- you don't need the internet to install dependencies. There are many options e.g., a locally cached tarball will do (no need to download the same file multiple times). Note: your source tree is not the place to put it (the same source can be built, tested, staged, deployed using different dependencies versions e.g., to support different distributions where different versions are available by default)
- if your build infrastructure is compromised; you have bigger problems than just worrying about dependencies
- you don't need to pull dependencies and dependencies of dependencies, etc into your source tree to keep the size of the test matrix in check even if you decided to support only a single version for each your dependencies.
As usual different requirements may lead to different trades off. There could be circumstances where to vendor dependencies is a valid choice but not due to the reasons you provided
> C, C++, Java, C# etc. programmers have been pulling dependencies in their repos for ages.
I'm not making this up: in my career, I have never worked on a project where this is the case, and I've worked for shops that write in three of those languages.
> I certainly prefer to manually update them from time to time than to have builds that work on my machine but fail for other people
That's your choice, and it's a little bit different because I'm assuming "other people" are end users -- those that want to recompile SumatraPDF from source for some bizarre reason -- not developers. Fixing a broken build is a skill that every developer should have, but not an end user. Once I learned how to write software, I never came across a situation as an end user compiling an open-source Unix package that I could not solve myself.
The opinion I'm sharing here is related to developing on a team, not distributing source for end-user consumption. It sounds like you don't develop SumatraPDF with many other people, either. Nothing like merge failures on a huge dependency that I didn't write to ruin a Monday.
Also, wait, SumatraPDF is built with dependencies in the codebase? What if a zero-day is discovered in one of your dependencies while you're on vacation for a month; what do distribution maintainers do? Sigh? Patch in the distribution and get to suffer through a merge when you return?
As for upstream maintainers rebuilding your package, I don't see how having to update a submodule is vastly different from updating the relevant versions in a file. Both seem like they'd take mere seconds.
It's not like you're writing gotos straight into library code, it's merely a bookkeeping change. You're just importing everything into your repo instead of referring to it by ambiguous version numbers. In the end, the code written and the binary produced should be identical.
The first time I worked on a C# project started in an age where nu-get was not widespread, I saw with dismay a "lib" directory with vendored DLLs. It does happen.
- Binary artifacts under version control is no-no for me, unless we're talking assets. Third-party libraries are not assets. - Where do these DLLs come from? How do I know it's not some patched version built by a developer on his machine? I have no guarantee the library can be upgraded to fix a security issue. - Will the DLLs work on another architecture? - What DLL does my application need, and which ones are transitive dependencies?
That's many questions I shouldn't have to ask, because that's what a good package management system solves for you.
Anyway, this analogy isn't helping anyone. You think libs in source control is a problem because some people might not do it properly. I'm contending that there's nothing wrong with libs in source control--there's something wrong with letting people who might not do it properly near your source control.
Another problem was that by building the way they had, they'd hidden the slowness and complexity of the build - including the same code from different branches via a web of dependencies, and with masses of unused code. They never felt this pain, so had no incentive to keep the code lean.
Which is why you pin dependencies to specific versions.
> - the host is down so you can't compile your own code because your dependency is not available
Which is why you have (caching) proxy and use mirrors.
> - the host is hacked and "foo" was replaced with "malicious foo"
Which is why you use GPG signing. (sadly, Python lacks in this respect).
> - exponential increase of testing (you really should test with all version of your dependencies you use)
Which is why you only use a specific version.
There's no such thing as, say, Maven binary dependencies for Java. If you don't check in the source code for your dependencies, your team member won't get the same version as you have and builds won't be reproducible. You won't be able to go back to a previous version of your app and rebuild it, because the trunk of your dependencies will have changed. By checking in the source code you're avoid a whole lot of hurt.
Checking in source is okay because Go source files are small and the compiler is fast. There isn't a huge third-party ecosystem for Go yet.
I'm saying there should be, but not necessarily the same thing. That's my entire point.
I'm also not a fan of the "you have a dissimilar opinion to mine, so obviously you've never used Go properly" attitude in this thread. One way to read your last is that I've never used Go at all, though I'm giving you the benefit of the doubt and assuming you meant used Go properly. Either way, I don't get the condescension of assuming I'm unaware of everything you're explaining to me simply because I have an opinion that is different than yours. Especially since half of your comment is repeating things to me that I said earlier.
The reason I assume you haven't used Go much is that your examples of problems with checking stuff in aren't examples of problems happening in Go. It's an analogy with other languages and other environments. Such arguments don't seem to get very far.
Maybe it won't scale and something will have to give. I expect the Go maintainers will find their own solution when it happens, and it won't look like Maven or traditional shared libraries. (If anything, it might be by replacing Git/hg with something that scales better.)
Disk space is cheap; much cheaper than time needed to fix something if it goes to hell and some remote repository isn't available.
Exactly this. This is just one reason I really like to "git clone --mirror" onto my server, then clone from that, and work from there.
Since I site the repos on a server I have online already, it is also trivially easy to make it publicly "pullable" for others if need be.
The real issue here seems to be that this was a half-finished project, at a very early stage of development, and the developer probably never intended their local scratch copy to be used in production [http://www.joystiq.com/2012/10/24/haunts-anatomy-of-a-kickst...]
If the original programmer is using git, he should have a complete history of his code. They know the date (or thereabout) when the last version of the code "worked". Revert down to the a commit around that date.
For each dependency, do the same. Pull the dependencies repo down, revert to the last commit that was around the date of the working version, and use a new fork of that repo to get things back to the working state.
One by one, move the dependencies forward, resolving bugs as you go.
git reset --hard commit-hash
where commit-hash is the last known working commit.
Just do that with all the repos around the date it last worked. If there are alot of commits between the last working version and HEAD it is easier to do this then reverting each commit.
Unlikely? Surprising? Insane? Sure. But never call something impossible unless you can not only write down a formal proof for it but also verify your assumptions.
The acid test will be if someone fixes their issue and says "Here you go, it's compiling again. Now just make sure you handle dependencies like this to keep everything working." Will the project be picked up again?
So you could rewind all the repos to where they were at a particular date/time with: `git co HEAD@{2012-11-26 18:30:00}` in each repo. If you run that same command in the project repo, and the dependency repos you should have a snapshot of the developers local checkout at that time.
HEAD could be a branch name or whatever ref. You can specify the time like ref@{2.days.3.hours.25.minutes.ago} as well, and these time specifiers work wherever you can specify a revision for example `git log -n1 master@{yesterday}`
see: https://www.kernel.org/pub/software/scm/git/docs/gitrevision...
My interpretation:
1. Original author writes a program using Lua. Author downloads the latest version of Lua into his system and includes and links it. Note that Lua is not checked into his version control system nor does he note the used version.
2. Lua upstream does a backwards incompatible change
3. Author extends his local (old) copy of Lua
Result: An external dev will find no combination of Lua version and program revision which compiles. The original author seems to have lost his local modified Lua version.
You can probably spot various mistakes an experience software developer would avoid. The interesting point is that Go seems to encourage Step 1 nonetheless?
One of the things I spent a bit of time on when Java was being developed was class management. We had an ongoing discussion about embedding versioning information in class files, creating signatures that could be checked at runtime, and just winging it. Bill Joy was convinced that it was possible to give the class loader the ability to statically analyze a class and determine whether or not it would work with the code that was trying to invoke it. This was when I was first exposed to the idea of creating a digest type 'signature' for a method/class invocation that would encode semantics.
The end result was 'just wing it' of course. This was expedient but didn't advance the state of the art :-)
Go's implementation seems a bit more dangerous. Not the least of which being drive by malware injection attacks on abandoned/poorly administered code repositories. It seems in principle to be no more or less dangerous than something like "apt-get install libcairo-dev ; make" but that would only be true if there was a trusted repository system.
Can someone from Go comment on the ability to create and manage trusted repositories? Should we look for a go 'distro' to build with?
Not that this is really a response one way or another, but given the way that Go currently works, you effectively place the same amount of trust in each individual library maintainer as you do with the collective trusted repository maintainers in other approaches.
If someone were to decide to start curating repositories and forking/maintaining them, people could choose to trust that individual. So the possibility for a trusted repository exists, but as far as I know nobody has fully stepped up to create such a thing at this point.
Here, 'work' must mean 'has the methods and other stuff the client code is looking for with the types the client code expects', except the Java type system isn't really expressive enough to do a very good job of that, especially if there are any generics involved (such that a lot of stuff has been hammered down to type Object). But Java didn't have generics that long ago, did it? Something like Eiffel, with design-by-contract and automated contract checking, would be able to make more use of the concept.
Pretty trivial, especially if you can rely on the compiler to do dead code elimination before checking to see what the client code is expecting. I don't know if javac (as opposed to the JIT) does dead code elimination, though.
The idea was to create a 'semantic message digest' (I don't know if such a thing exists or not) which could be computed at class load time and checked/validated at method invocation time. The canonical case for the talking point was you have a class Foo with methods doA, doB, and doC if you change doB incompatibly but the client only calls doA and doC the class is still 'compatible' (in a call contract context) with the class even though the class is 'different' in an incompatible way. Guy Steele and Bill came up with a pretty interesting list with things like methods added, methods missing, more parameters, fewer parameters, additional fields, fewer fields, additional constructors, Etc. All the ways you might mutate a Class and yet for a class of callers it would still work just fine.
If one can do this, another interesting feature is better capability type systems. Those systems have to deal with changes that may change the capability represented by a method even though it is still semantically compatible.
- it comes from a repository that has been signed by the distributor.
- the package has been tested and has a good level of integration with the system (ie different distributions may package things differently).
- depending on the distribution, there may be a commitment to API/ABI stability (ie. Debian backports security fixes most of the time).
That said... running "go get" is very convenient if you control the code repository where you're fetching the code, but it's not a good idea when using 3rd party software that is in active development.
EDIT: formatting
You basically just need to throw up a HTTP-accessible HTML page with some <meta> tags that describe how to fetch the actual code with a variety of source control systems. Anyone can create their own trusted repositories.
[docs] http://golang.org/cmd/go/#hdr-Remote_import_path_syntax
I agree that, if this problem had been foreseen during development, better version control could have helped.
> Go didn't screw anything over
This is where I disagree. Having a language feature to fetch dependencies from mutable online sources means that you're practically asking for programs to break unpredictably based on external changes. The local cache makes it worse, because every machine will have a slightly different version of the library, and bugs won't be reproducible.
Do I understand you correctly?
Yes, you are trusting Google not to maliciously change the file to something evil, but that is quite different to linking to the trunk of a repository that is expected to change.
Visiting http://golang.org/doc/code.html I see examples like this:
import "code.google.com/p/go.example/newmath"
Again, as I mentioned elsethread, I'm pretty new to Go, so perhaps I've been misusing it.
How would you guarantee that known version? Using the normal import above would not do this, esp. if you are working with other developers or have to switch computer/reinstall your dev environment/update. You'd get the current head (not a known version) of all dependencies whenever you run go get on a fresh install, because there is no package versioning in go import statements or go get.
The only way currently to guarantee you get a known version is to fork your dependencies, put all the code in your own repository and update them manually. That's not the end of the world but it's not as elegant as the rest of the go build system.
Because that's the source code sitting in your src tree. That's actually exactly what 'go get' will do: pull the source code of the current version of the dependency into your src tree and build objects in your pkg tree.
> The only way currently to guarantee you get a known version is to fork your dependencies, put all the code in your own repository and update them manually.
Nope, you just use (e.g.) a git submodule for the dependency. When you need to, but only when you need to, you can update each dependency or all at once.
Using go get on its own is not enough to end up with a known version - each go get can pull different code, unless you have manually set up dependencies first as part of the package or checked in the entire src tree and shared that. This works fine for one person working on code but obviously requires a bit more work if you're sharing code with dependencies with others.
I not trying to say it's impossible to manage or a huge flaw in go get, but it does require you to deal with dependency versions explicitly yourself, unlike many other packaging systems.
That's what I was describing. The developer of the dependent code pulls in the dependency with go get, and then other developers see both his dependent code and the dependency he was using.
No it doesn't. Go get or import doesn't let you specify a tag of a repo - if it had the developers would have an easy way out of this situation as they could specify their dependencies on import when first importing. That's not to say this is the go developers fault, but it does highlight a weakness in go dependency management:
Go get always checks out the HEAD. They have a scheme for checking out tags only for golang versions so they have the code to handle tags etc in popular repos, but hardly anyone uses the language versioning support because it just isn't (and shouldn't be) required, and requires a specific scheme of tag names to work anyway. At present golang versioning is only used for this, and it's not widely used as a result.
Of course you can work around this, and it's hardly a showstopper - at the simplest level you could go get then checkout a specific branch with git instead before building, or fork the repo, keep local copies in your source control etc. you only have to do this once for dependencies and then they won't change unless you decide to update them.
The bizarre thing is that go get does know about git tags and versioning it just refuses to make that useful for users unless they want to version the language, not the libraries they use - I imagine this was useful in early language development, but now that they've hit 1.0 and are promising to be backwards compatible it seems pretty pointless.
I'm not sure why you think that. It's perfectly reasonable to want to use new features in Go 1.1 while maintaining backward compatibility (or an older version) for Go 1.0.
I haven't seen much use of the go language auto-versioning feature in the wild, whereas lots of people have noticed the lack of support for versioning packages/imports and the implications for unpredictable builds - any dependency imported the normal way could break at some time in the future or for different coworkers who checked out at different times, unless you fork which has its own problems for maintenance. Personally I think versioning packages is more important, and would prefer to have that option rather than the option to maintain branches for different versions of go (which can be done manually where really required and is unlikely to be an issue in the 1.x series).
The features have nothing to do with each other---it's not an "either or" deal.
I haven't used the `go1.0` or `go1.1` tags myself.
People have tried setting up versioned packages, but it hasn't taken. As for me, I'm quite happy with the simplicity of `go get`. It's been working great for me for over a year.
I also like the simplicity of go get and don't feel this is big deal, but having used other packaging systems suspect that go will start versioning packages eventually as it makes life a lot easier if you are building on code from elsewhere and want to share code. The present setup works fine, with the caveat that if you don't fork all dependencies, other users might see different build results for the same code.
I strongly disagree. I've used other packaging systems and they are a constant source of grief. I very rarely need to pin dependencies so it doesn't make sense to work with a super-powerful tool that allows dependency management at the version level. If I did, then I understand the need for the power.
> The present setup works fine, with the caveat that if you don't fork all dependencies, other users might see different build results for the same code.
That's not a caveat; that's a strength. When this happens, I get a bug report from a user, and I fix my software. Then it works.
Bitrot is a damn hard problem to solve. I'd rather it smack me in the face then creep up on me and strike when I least expect it.
N.B. I fully understand there are scenarios when reliability of builds is important. If I were in that scenario, I'd accept the pain of pinning dependencies or writing my own tool to do so. The great thing about Go is that I could do that with relatively little pain. (Its standard library contains amazing tools for analyzing Go source files.)
My links to documents on SGI don't seem to work...
An equivalent would be linking to a HEAD version of a library.
Go devs always said that Go is designed to solve Google problems in Googley ways. All Google code builds from HEAD, at least it did when I was last there. The reasoning for this is that breaking changes can be discovered ASAP, instead of depending on code several versions back that might be much more painful to upgrade should the need arise.
This is another case of the Google way having a sane reasoning, but perhaps does not work in other cases. I'd personally prefer some versioning system, but I can see why they've done it the way they did.
Looks like Google should not advertize Go as general purpose language for everyone to use? Especially the tone of Go related postings here on HN is that everyone should replace C, PHP, Python, Java, JavaScript, etc. with Go because it's better, faster, more scalable on server.
Erm, yes it did. The civilized world (Java, Ruby, Python, Clojure, Scala, Haskell, OCaml) has version numbers in their dependency management. Albeit out-of-band from the source files (pom.xml, gemfile, requirements.txt, project.clj, sbt, .cabal, OPAM version pinning) but it does work.
Hell with Clojure and Leiningen (the standard choice for dependency mgmt) even the language version is a per project dependency ala:
[org.clojure/clojure "1.5.1"]
So you can still build jars that "just work" even if it's some legacy stuff that you haven't updated to the latest Clojure version yet.
That's not to say Clojure would've helped here - it's a game. And one that's aiming for better graphics - JVM isn't a good idea.
But that doesn't excuse Go's poor package management design that decided being facile and hid complexity of the real problems (changes breaking existing code) was more important than working builds.
The decision to make it facile, weak, and fragile was egregious and very out-of-character with the rest of their design decisions. I think Go can be faulted for failing at their own goals on this particular matter.
use Moose 2.08;
I'm not a Perl user, I don't consider running sed on my source code to be a "bonus".
Still better than Go though.
That was completely unnecessary.
M4? Make + C preprocessor? Perl generating Perl? I tried to guess the least-absurd option for updating a library.
Please inform the rest of us what the standard tooling for handling this in the Perl community is.
I'm sure Perl isn't unique here, of course.
Go has localised dependencies: you package your code with exactly the versions you need.
> So you can still build jars that "just work" even if it's some legacy stuff that you haven't updated to the latest Clojure version yet.
And with a Go project, one would have a complete system which 'just works,' complete with all dependencies, regardless of how old those dependencies actually are.
> But that doesn't excuse Go's poor package management design that decided being facile and hid complexity of the real problems (changes breaking existing code) was more important than working builds.
Go doesn't do what you think it does (probably not your fault: the article is misleading). It only pulls down updates if you tell it to. It sounds like the developer of the original code was doing the wrong thing, not using GOPATH the way it was designed, not checking his entire source tree—including dependencies—into version control. I could be wrong, of course: it's possible that he really did use it properly but discovered a misfeature I've not yet found.
I think the fact that every Go proponent attacks the developer of the game is one thing which keeps me from using Go. The community seems to be incredibly closed-minded and hostile.
Nah, not that I like Go very much but while reading this article I was like "oh, and how is this Go's fault?". It is like I'd depend on some library and then the library gets pulled from the internet or updated in an incompatible way, how is this a problem of the language that I develop in.
I might as well blame my text editor for allowing me to do stupid things.
The difference is whether the editor turns around and deletes /home as a punishment (Go community) or tries to educate the user about a better approach without acting like an asshole (pretty much all other language communities).
From the perspective of one developer's checkout it can't. But a new clone of the project since dependencies have moved yields a new product where the libraries have "moved" since a previous clone. I think that counts as "changing underneath you" for some definition of that term.
That being said I don't really think it is Go's fault for the developer not understanding that they need to keep a clone of dependencies if they don't want them to move.
No, because a project's tree contains the source code of all dependencies. Why do folks keep on saying this?
Here's how it works: you create a directory foo-proj, then export GOPATH=/path/to/foo-proj:$GOPATH (or whatever you like); then you run 'go get github.com/baz/bar example.org/quux'. Go downloads the current version of the bar and quux libraries, and then creates foo-proj/src/github.com/baz/bar and foo-proj/src/example.org/quux, putting the right files in the right place; it then builds each package, putting the object files in foo-proj/pkg.
As the developer, you configure your VCS to ignore foo-proj/pkg, then you commit foo-proj. You might put your own code in foo-proj/src/fooproj, or foo-proj/src/example.com/foo or whatever.
When another developer clones your project, he gets foo-proj, which includes foo-proj/src, which contains foo-proj/github.com/bar/baz and foo-proj/example.org/quux and everything else.
I suspect what happened in this case is that the developer was building his code as another library, rather than as a project, and pulling it into his GOROOT, letting Go grab his dependencies but not putting them into version control.
It's also possible that he was doing the right thing, but didn't realise that git wasn't tracking submodules without his involvement.
It _does_ do it by default; the problem in this case appears to be (although I could be wrong) that the developer went out of his way to do the wrong thing. As I've noted elsethread, I could be wrong; possibly he was misled by something else.
But the default way Go handles dependencies is exactly the way one would expect: you code against one version of a dependency, and have to manually pull in a new one.
https://groups.google.com/forum/?fromgroups=#!topic/golang-n...
Yes and it is only the fault of these devs that they did not do that. One of the first things I did when starting a job programming Go full time was go- woah, we need to clone our dependencies so we can manage upgrades. It was just common sense and easy.
Sample output:
# github.com/runningwild/glop/gos
/usr/bin/ld: cannot find -lglop
collect2: error: ld returned 1 exit status
# github.com/runningwild/opengl/gl
gl.go:142: cannot use _Ctype_GLint(mapsize) (type C.GLint) as type C.GLsizei in function argument
gl.go:147: cannot use _Ctype_GLint(mapsize) (type C.GLint) as type C.GLsizei in function argument
gl.go:152: cannot use _Ctype_GLint(mapsize) (type C.GLint) as type C.GLsizei in function argument
gl.go:158: cannot use _Ctype_GLenum(internalformat) (type C.GLenum) as type C.GLint in function argument
gl.go:164: cannot use _Ctype_GLenum(internalformat) (type C.GLenum) as type C.GLint in function argument
gl.go:170: cannot use _Ctype_GLenum(internalformat) (type C.GLenum) as type C.GLint in function argument
/usr/bin/ld: Warning: size of symbol `luaX_tokens' changed from 8 in $WORK/github.com/xenith-studios/golua/_obj/lcode.o to 256 in $WORK/github.com/xenith-studios/golua/_obj/llex.o
/usr/bin/ld: Warning: size of symbol `luaT_typenames' changed from 8 in $WORK/github.com/xenith-studios/golua/_obj/lapi.o to 88 in $WORK/github.com/xenith-studios/golua/_obj/ltm.o
# github.com/xenith-studios/golua
/usr/bin/ld: Warning: size of symbol `luaX_tokens' changed from 8 in $WORK/github.com/xenith-studios/golua/_obj/lcode.o to 256 in $WORK/github.com/xenith-studios/golua/_obj/llex.o
/usr/bin/ld: Warning: size of symbol `luaT_typenames' changed from 8 in $WORK/github.com/xenith-studios/golua/_obj/lapi.o to 88 in $WORK/github.com/xenith-studios/golua/_obj/ltm.o
They should file an issue on GitHub.
Or you could create a version of your src folder as a github checkout with submodules for each of the things you want to import.
I struggle to see what the issue is. I knew the first time I wrote an import in Go that this was possible, and that as the libraries for Go are relatively young that it is also likely.
My personal preference is just to keep track of changes fairly regularly and not allow much drift. But that's because up until Go1.1 I was following tip, which meant I was already in that habit.
Though it would be nice if ``go get`` could take a revision or branch/tag name.
"Original author didn't localize dependencies, obviously caused problems."
"I don't know anything about Go but I have incredibly strong negative opinions about it based entirely on this article. Oh also, I hate GitHub."
I'm a little suspicious when it is claimed that the original author can't get it to compile. That is just so strange and the simplest explanation was that it was never in a great state anyway.
In retrospect they were probably being protective of their IPR, but erred on the side of being overprotective and pushing away a few folk that could have helped.
Maintaining software is hard.
Blaming your own incompetence on a library or an entire language is easy.
I've been guilty of this at times, but when you get down to it, it's your job as a software developer to understand how your software works and how to maintain it well. This doesn't just include the code you write; it includes your tools, third party libraries, and the internals of the language (VM, compiler, etc.). Its not easy, but it's part of the job.
On a more positive note, versioning packages in Go is already possible and is starting to gain momentum. Check out http://www.gonuts.io/
And it's the job of language (and language ecosystem) developers to make this as straightforward as possible. It's perfectly possible to blame both the author and the language.
The developer should have set up a proper Go tree for his project, prepended it to GOPATH, run 'go get DEPENDENCY1…' (which would have put the complete current source of that dependency into his src directory) and then checked his project tree into his VCS (adhering, of course, to the licenses of his dependencies). Anyone who downloaded his code would have gotten the compatible version of all dependencies from the VCS. Then when he felt it was time to upgrade the dependency, he should have committed his work (and branched in something like git), then run 'go get -u DEPENDENCY1…' to fetch the latest version, which—yes—would break his code's compilation. The next step would be to fix his code to work with the new version, then commit that.
It sounds like the guy was doing something developing his project in his main Go tree, not putting his dependencies into version control, blindly updating dependencies and so forth.
But possibly I missed something.
If the original developer does not have that but at least hasn't nuked his src directory, he will have all of the working code (but may need to, e.g., git bisect).
Go's sin here is being permissive. And it's true that clearly the game dev team just... didn't do things right. They were pretty much on a collision course with failure by managing their project's code the way they did. But Go allowed them to get much further along than they otherwise would have.
There are some issues with the way the Go tooling handles dependencies, but I think the minimalist way in which it does is very useful, and easy to build upon. Go uses a distributed dependency system, and you can't give it the same level of trust you give to centralized repositories, like one does with gems, Pypi, NPM, CPAN, and so on.
I've written complex OpenGL ES games from the ground up before, and its a long slog, especially at the end where you you are trying to crank it up to 60 FPS and doing tons of work rewriting it to be high performance.
Managing DLL hell may be acceptable to ship a game, but it's not acceptable to openly develop software.
In this case it wasn't even intended to be open-source, which may be why the dev went the route he did, but doing so essentially closed the door on the later option of easily letting the community take over on code development.
There's a reason people codified semantic versioning. Projects like gstreamer take it even further and make the major version part of the library name, even allowing for co-installability of old and new (e.g. gstreamer-0.10 and gstreamer-1.0 can both be installed and used).
Why not? Having something that compiles and runs is probably a bit of a boon to the development effort.
It's not acceptable to merely "manage" dependency hell as the net result is still unmanageable at OSS scale. You have to essentially eliminate the interplay of "which exact version do I have" with your code, which is why things like semver and soversions (on .ld.so's) try to accomplish.
Being able to have an unknown user download your code and the listed dependencies and have a repeatable build is invaluable for development.
Go's dependency management will even allow a library to have multiple versions depending on the version of Go being used to build (either a branch or tag matching the Go version). This isn't a major feature right now, since Go is backwards compatible within major versions, but could be much more important when there's both go1.x and go2.x in production.
The problem here arises when your remote library can't maintain a stable master, either because the maintainers don't care, don't know how, or the project is new and still in flux. The onus is then on you to incorporate that code into your project.
People need to think of this system as giving all dependency developers commit access to your project (which is nearly is). Would you just let the world check-in code willy-nilly, or are you going to review what's going in?
Whenever I see a fast-moving dependency I fork it and use my fork, then integrate upstream changes when appropriate. If you have to do that for every damn dependency I can see that being horrible, though...
If you keep your system simple and don't rely on too many "kitchen sink" frameworks, then keeping your own source tree of your dependencies isn't that much different from maintaining a local repository of the same code in compiled form (with the added benefit of having the original source available).
Go doesn't have a gem/pip/cpan/npm/... like ecosystem, instead you import git/mercurial repos directly. This is pretty nice for rapid development and to test things out, but not the right approach for complex long term projects like a game.
The original author of the game missed to include the dependencies into the project and now they are doomed. Put the libs you rely on in your version control, be it as a submodule.
I'm wondering if anybody is working on a package manager for Go libs... That would be great.
@aleksi is: http://www.gonuts.io/
I don't think it'd be difficult to create a CGAN/GoPI/Gobal/gojars/etc. that provided versioning.
Starting to build something big with tools you are not yet very familiar with is always risky.
If the game was finished and working but does not compile anymore, that sounds like a trivial thing for a hardworking developer to fix in a pretty short amount of time.
I've seen much worse when porting games across platforms. This doesn't look even close to "beyond repair".
Also, if you want even more control(ie, you want to modify the library you're using in some way), fork it on github and include your fork as a submodule. That way you can patch bugs easily(and send the upstream a pull request) without having to wait for upstream to fix things you know how to fix yourself.
I can tell that the article and its comments is absolutely misleading. Seems that the developer(s) are trying to justify the lack of professionalism and blame language/tools/someone else. In the article I see total misunderstanding of how the Go ecosystem works, strange decisions, obvious failures during development and weird situations like 'original dev cannot compile the code anymore'.
I think an engineer should have a decent understanding of the tools/concepts/... BEFORE starting to use them. Not AFTER some problems occur.
> Why does this matter?
Maybe I'm not understanding this, but the dependency issue and the issues in this paragraph are not related. Issues like improper syntax? That's not related.
Sounds like bad planning as far as the dependency stuff goes. For everything else, sounds like a distribution issue.
I assume, instedad of forking, it might be possible to import a specific tag, or commit hash from a git repo. If it's possible, it'd solve the backward incompatible change problem but I guess the original author can always delete the github repo.
What is a good way to manage dependencies locally?
What I'd like to see is a site similar to this, with a similar tool. Instead of having to upload the package to the site though, you could add a `pkg.json` file (or something similar) with application data. Every time this is committed, the site would automatically check to see if the version has changed, and then index that commit hash as a new version.
This almost sounds fun...
What I really want to know is why they chose Go in the first place. Go isn't a game development language and it has little to no support, documentation, or just plain old previous community experience in the gaming context. This feels like the classic software development of putting the technology ahead of the product - using a tool because it's cool, rather than using it because it's the best thing for the job.
It's all about shipping, people. Use whatever you need to you can ship what you want, when you want to, at the quality and reliability you desire.
In the C and C++ world, yes. Because they lack the software repositories many mainstream languages have.
The problem is that we got used to Maven, Ivy, CPAN, Gems, eggs, NuGet, ...
As I said in another comment regarding Go. If any company has the ability to create a stable, maintained CPAN / rubygems equivalent for the Go language, it would be Google.
Which just makes the lack of it more jarring (although I see the arguments that Go is designed for Google and may not fit other's ideas of what important features are)
I realise that since Go supports this, it is encouraging to simply do that, but it isn't wise in the long term.
For instance, I've had some issues with Go-SDL, because one of the forks became unreliable. I decided eventually to simply fork another of the forks myself to overcome all these issues, and simply update my fork with commits to the main project whenever they came around.
Using an unofficial release or development version of a library is always risk. It is not language dependent. Things can always be broken if you don't follow release notes on an actively developed project.
Back on topic, maybe Go should take a page from node's NPM: simple versioned tarballs with isolated dependencies.
I don't see how bashing Go would help these guys out of the hole they are in, same would have happened with any other programming language in a project that uses fast-moving dependencies...
They were making a game, they could have just forked the projects they needed, and build from that, that's the way to do these kind of projects, you stick on a revision, and when there is an update on a dependency, the team should agree to make the upgrade changing all the required source code.
For what they say, the main developer had no idea that the repositories were changing a lot, and he never updated them, so everything was going thru smoothly... I don't see the need to bash Go, neither the main programmer, i just believe he needed more experience to handle the project correctly.
What i don't like is their attitude "nobody can fix this Go garbage code because Go is all wrong"...
Gotta love the Internets.
import "github.com/go-gl/opengl/gl"
I'll take a stab at it this weekend (if I have time).
For the curious, here is their repo: https://github.com/runningwild/haunts