A little Golang way
aerofs.com
aerofs.com
I know it seems like hype, but I encourage others to try Golang out, maybe even slap a web app together with it. What you'll find is that your Go app will be extremely efficient and performant. It's kind of unfair to compare Go to many of the other languages of the web because it is compiled, which is a huge reason why it is so performant. So rather than compare Go to other languages, I encourage you all to give it a try. If you already program, working on a small Go side project will get you up to speed quickly and you'll learn about some of the awesome packages individuals from the Go community have put together for us.
Disclaimer: I do not work for Google. I write Go code and enjoy it. I think others will too.
This is true, for sure. But my experience--and I've written plenty of Go, though I find it unpleasant to write and think about for all the usual reasons that Go partisans roll their eyes and so would rather deal with it as artifacts rather than actually writing it--has led to watching lots of developers make mudball codebases in the process (while blogging very heavily about how great it is three weeks in, and less a year-plus in). A pursuit of faux-simplicity has led to what I see as reinventing a lot of Java-1.4-era (because the language itself is essentially that) design patterns--and we should be reminded that design patterns exist to address defects in tooling--that more expressive languages have managed to avoid; the general desire for smaller applications helps to some extent, but software has this tendency to grow that I don't think Go gives you the tools to effectively manage and maintain. Reasonable minds can differ, of course.
Past that, while I think it's an alright choice for company-internal deployable products--web servers, worker nodes, etc.--I have straight-up problems with it being the new devops tool of choice. Statically linking your SSL library makes you an asshole when it inevitably fails and now an application has to be regression-tested so that new features and new bugs don't hose you just because that's the only way to upgrade Heartbleed 2.0. (Go also discourages program extensibility through components and rather as recompilation; the stuff Packer and Terraform do to provide "plugins" that the same company's Vagrant just did with a `require` is gross and, to my mind, completely foolish.)
So I wouldn't say it's hype, but I would say it's not all true, either.
Can you elaborate on this? Because in my experience with Go, I found that I had to make _far_ more components (assuming this means libraries?) than other languages.
> Statically linking your SSL library makes you an asshole when it inevitably fails and now an application has to be regression-tested so that new features and new bugs don't hose you just because that's the only way to upgrade Heartbleed 2.0
Not sure I understand that last bit. I get that statically linking could be bad(ish) because you would need to know to recompile with a fixed library, but what was that about new features?
With shared libs, you can keep using an old version if it works for you, while still updating ssl to a fixed version (assuming api compat).
And not just new features--though SemVer is very often honored in the breach more than the observance--but breaking, sometimes undocumented changes (two Go projects, Packer and Terraform, both come to mind).
The same can be done for a Go app and it'll be available via apt just as any other app.
A fair comparison would be compiling a C/C++ app from sources vs. compiling Go app from sources and Go wins that contest easily.
It is also much more likely that a "minor version bump" that happens to contain the dependency with the bug has regressions or new, untested-in-my-environment features that I must accept as the price of a Go application's upgrade unless I want to start playing with said application's vendored dependencies. Which I don't, which is why libimportantthing3 is a vastly superior choice for software I must use but do not want to adopt and care for.
It's an unrelated field to my day job of devops/server software, but I wrote a plugin system in .NET and it's super trivial to just suck down an assembly and expose its types to the core logic. I've written the same in Java, and Ruby basically makes it a breeze with a Gemfile and a 'require'. It can be slower (though for the overwhelming majority of tasks not at all too slow for a tool, rather than a high-throughput server or whatever) than Go can in some cases be, but it doesn't suck to use, and it's for that reason that while I can understand Go for server stuff I have a real beef with it intruding on my systems when I need to use it as a user.
This is a really important point, and I think it's why a number of best-practices in Go are actually not best practices in other languages, and vice versa.
One of the explicit, top-level design goals of Go was to focus on creating top-notch tooling as part of the language. While it is not the only language that has tried to do this from the get-go, it's one of a very small number[0].
Because the tooling was a first-class design goal, a number of the problems that traditional design patterns were created to address are less problematic in Go code[1].
[0] Case-in-point: gofmt, which other languages are now adopting due to its success.
[1] Again, gofmt: there are a number of design patterns and style bikesheds around code format, but really, the most important thing is that there exist a uniform standard. gofmt provides that reliably and a way to enforce that as a pre-commit hook, which has done wonders for eliminating minor style variations that IMHO cause more problems than they solve.
> [1] Again: gofmt…
What does gofmt have to do with design patterns?
gofmt is about code formatting. Design patterns are about abstraction and expressiveness. A code formatting tool does nothing to address abstraction and expressiveness of the language.
First, gofmt can do more than code formatting, such as applying simplifying code transformations that are semantically equivalent. Second, code formatting and design patterns are indeed related, because the way code is laid out in text is the way that design patterns are expressed. The layout of code affects how abstractions are presented, which affects which abstractions are easy to reason about and work with.
Finally, I picked gofmt because it's a pretty uncontroversial tool, and one that is so successful that even languages like Rust have adopted or are working something similar. I'm really not interested in starting another flamewar about why Go lacks $FEATURE and therefore $OTHER_LANG is better, because we have had enough of those on HN, don't you think?
Which also has nothing to do with design patterns. (At least not if you're limited to the extremely basic -r gofmt rewrite rules.)
> Second, code formatting and design patterns are indeed related, because the way code is laid out in text is the way that design patterns are expressed. The layout of code affects how abstractions are presented, which affects which abstractions are easy to reason about and work with.
No, I don't buy that a code formatting tool obviates the need for design patterns. How does gofmt replace the Visitor pattern (just to pick one at random from the GoF)?
This would not be the first time that we have had a discussion on HN about this exact design pattern in Go, so it's hardly a random choice. And given past precedent, I think it's best if I end my half of the conversation here and propose to agree to disagree. To quote my previous post,
> I'm really not interested in starting another flamewar about why Go lacks $X and therefore $Y is better, because we have had enough of those on HN, don't you think?
Asking if a code formatting tool "obviates the need for design patterns" is the wrong question, because it assumes that there is a need for design patterns in the first place.
I do agree with you that a code formatting tool is not capable of somehow fixing software design and architecture decisions. However, "design patterns" by their GoF meaning exist as to address shortcomings of the languages and tools used: most of their advice does not make sense as soon as you move away from Java into less object-oriented or less procedural languages. To talk about "the need for design patterns" as if they are some sort of mathematical truth is misleading and dangerous.
So, yes, I would argue that Go does not need the Visitor pattern (and so would Rob Pike [https://groups.google.com/forum/#!msg/golang-nuts/3fOIZ1VLn1...] ), code formatting tool notwithstanding.
In that message, Rob Pike misunderstands what the visitor pattern is for. Go's "type switch" is just chained Java instanceof. Java still benefits from the visitor pattern for (e.g.) compiler transformations, even though it has instanceof.
Languages differ in which design patterns have a native expression in the language, which can be reduced to simple reusable library code, and which require type-it-in-each-time fill-in-the-blanks code recipes.
The fact that Design Patterns were popularized in software development by the GoF book which, among other things, included code recipes to illustrate its patterns, and that the patterns in it tended to be ones for which the popular languages of the day required code recipes (lacking native implementations or the ability to provide general implementations as library code), has unfortunately associated the term with the code recipes, which aren't really the central point of understanding and using patterns.
Did you all write your code in notepad before?
I didn't mention Java 1.4 just for a lark; it's been long enough since I used it that I don't remember the ecosystem well but I do remember the style of coding being so brutally centered around type assertions and blind casts that Go really does remind me a lot of it.
Could you elaborate on the static analysis that you find lacking?
The debugger was usable. You had realtime memory/CPU profiling tools like JProfiler. There was a workaround to get JMX working (JMXRI) with is great for production monitoring. Code formatting tools were significantly better than gofmt. You had bug finding static analysis tools e.g. Findbugs which integrated well into build tools.
And of course you had great IDEs like Eclipse (which was actually great back then), NetBeans, JDeveloper which allowed for refactoring, autocompletion and code assistance.
I'm working with Golang every day now and the ONLY reason I'm using it is its CSP concurrency model, which is still way behind what Clojure's core.async can do.
Go has its own SSL library, crypto/tls, which is not linked to any C libraries and wasn't affected by Heartbleed 1.0. You haven't written much Go if you don't know that.
The argument is specious anyway, there's nothing difficult about building a binary from an old version of your code or just upgrading in most cases. Deploying a new Go binary is always trivial as compared with upgrading, testing, and deploying Python, Ruby, or Java applications with their associated libraries and interpreters.
The parent poster obviously wasn't saying that crypto/tls was affected by Heartbleed specifically. It was a statement about the security implications of static linking.
Is it going to be affected by Heartbleet 2.0? How about whatever the next exploit is?
"The argument is specious anyway, there's nothing difficult about building a binary from an old version of your code or just upgrading in most cases."
Not in every case, and not for customers who don't have access to the code.
Really? Java 1.4 had an extensive and consistent standard library, implicit interfaces, composition instead of inheritance, static builds, and built-in concurrency?
Go is not early Java, it's more like C 2.0, and I don't really see anything faux about the simplicity, it really is pretty simple, perhaps too simple for some tastes, but it's not pretending to be simple, nor is it simplistic. Which specific design patterns did you have in mind?
yes
> implicit interfaces
no
> composition over inheritance
yes
> static builds
no
> built-in concurrency
sure, with an 1:1 thread model as opposed to an M:N thread model.
M:N is not always a clear win over 1:1:
https://mail.mozilla.org/pipermail/rust-dev/2013-November/00...
http://xiao-feng.blogspot.com/2008/08/thread-mapping-11-vs-m...
Also, to be very clear, Java's threads ("green threads") were _originally_ M:N but they switched back to 1:1.
I think the standard library is debatable (see the infamous early Java Date class), Java encourages using inheritance (so not composition instead of inheritance) - something Gosling later remarked as his biggest regret, and concurrency had improved tools in Java 2, not sure the story was as good in early Java.
Java's "green threads" were N:1 threads, not M:N threads. While both are sometimes referred to as "green threads", they are very different things. (N:1 tends to give cheap concurrency but no parallelism, M:N tends to give cheap concurrency with as much parallelism as the hardware can handle, but with more overhead than 1:1 threads.)
You can do Erlang-style concurrency with 1:1 or N:1 threading models (and there are libraries for languages whose many implementations are N:1 or 1:1 rather than M:N that do that), but it makes most sense when you have M:N.
A modern Linux kernel has excellent thread scalability, and you can customize thread stack sizes to improve memory usage. (Thread memory use is kind of independent of 1:1 vs. M:N, honestly; the thing that can improve scalability is relocatable stacks, which is really not the same thing.)
It's instructive to look at the history of M:N versus 1:1 in the days of NPTL and LinuxThreads and compare that to the history of programming languages. In that world it was received wisdom that M:N would be superior for the reasons always cited today, but at the end of the day 1:1 was found to be better in practice, because the Linux kernel is pretty darn good at scheduling and pretty darn fast at syscalls. Nobody advocates M:N anymore in the Linux world; it's universally agreed to be a dead end. Now, to be fair to Golang, relocatable stacks do change the equation somewhat, but (a) as argued above, I think that's really independent of 1:1 vs. M:N; (b) any sort of userspace threading won't get you to the performance of, say, nginx, as once you have any sort of stack per "thread" you've already lost.
Cross-platform code can't count on always running on a modern Linux kernel, though.
But, sure, I'd think that M:N (which is never free) is going to be less likely to be worth the cost if you are specifically targeting an underlying platform reduces the cost of high numbers of native threads and the cost of native thread switching so that the price for user-level threading in the runtime isn't buying you improvements in those areas.
My mental model, which probably comes from everyone complaining about apaches thread-per-request model a few years ago, is that you basically either consume lots of resources with lots of threads, or you write ugly cps style code. I view haskell/go as giving the best of both worlds, but perhaps they aren't separate worlds after all.
The choice of 1:1 and M:N does not affect the programming model at all. It is simply an implementation detail. M:N scheduling is no more a requirement for CSP than Unix is for TCP.
In Erlang you don't think twice about spawning a million threads if that models your problem nicely. The same isn't true when you're dealing with kernel threads. (Right?) So I'm interested in green threads because when I have to start thinking about the cost of the threads I'm using, then I'm thinking less about how to most elegantly solve the problem and more about how to satisfy the architecture I'm programming on.
Now, if kernel threads are massively cheap these days and there's no problem spawning a million of them, then I need to take another look at that model.
Stackless coroutines are great--you can get to nginx levels of performance with them--but they aren't M:N threading as seen in Golang. Once you have a stack, as Erlang and Go do, you've already paid a large portion of the cost of 1:1 threading.
That's usually the case. Coroutines have their uses, but having used goroutines, that is my current preference.
Java: http://docs.paralleluniverse.co/quasar/javadoc/co/parallelun...
Clojure: http://docs.paralleluniverse.co/pulsar/api/co.paralleluniver...
We'll have a nice slect API for Kotlin, too, very soon.
Or perhaps Go is C+
I get what you're saying, and I agree with some of the criticism around the tooling, although I do think much of that owes to the age of the language. There are some specific things in the (sometimes rather cranky) replies to you that I think are incorrect but I'm not going to debate the merits of the language. Use it if you like, don't use it if you don't.
I will say that we've been using Go for nearly all back end services for nearly 3 years now, and we all still think it's pretty great. Our code base has also remained pretty clean, and in fact we're investing more heavily in Go going forward.
Pretty much the opposite of the mess that was and is Java.
Yes, Go is technically compiled, but the development cycle is closer to that of dynamic languages. In a similar vein, I have a lot of trouble taking criticism of Go's type system seriously. Is it as robust as Rust's? Probably not (I've never used Rust), but that's way beside the point. Go's type system gives you a great deal of benefits while keeping the language very dynamic-y.
tl;dr: haters gonna hate. I like Go very much for certain things.
Plus thinking is an enjoyable and rewarding experience.
Either way, the elimination of long compile cycles and other interruptions maintains the mental flow of the programmer. This results in a significant productivity boost. Also, the job feels better as interruptions and backtracking can be stressful.
I think Go sits on the opposite end of the spectrum. It's a hacker's language, not a mathematician's language. It's ugly and very useful.
Lmao. That's exactly what hackers said about LISP. That people can often emulate a language or even paradigm (eg OOP) within the original language using macro's shows it. Strange enough, LISP can modify itself to do whatever modern languages are doing with similar productivity. It doesn't work vice versa. On top of it, the LISP compilers make pretty fast code for such a flexible language and past engineers even made dedicated hardware for it.
That's why, although far from perfect, LISP is still the ultimate, hackers' language.
Lisp is a great language. This wasn't meant to be a jab at lisp.
Julia seems to be the best example of a clean-slate language that tries to combine all the best features and attributes of various languages with good effect. Also beats Go and many other languages on various benchmarks despite dynamic typing. I'd like to see some application-server or RDBMS type of benchmarks, though, as Julia was designed for the mathematical stuff. Might not perform as well but should still be good.
The runtime is shaping up quite nicely. Given dynamic linking and this language can pretty much pull its weight in the backend for almost anything.
Originally it was meant to be just a quick kick-golang-tires prototype, to be replaced with C++ later, but it turned out to work really well.
My impression of golang is that it's very useful getting things done. The code has also been readable to other people pretty much immediately, and they've been able to contribute features into it. None of us used golang previously.
Had I written same in C++, I bet there'd been a lot more bugs to fix.
Web apps are on the decline, being gradually replaced by Android/iOS apps. When people can use Go to slap up a performant Android app, it may find more use.
Building native apps is hard unless you are specialized , but puking up some html and json is easy ,hence building a blog is usually good for introduction.
The web is not going away anytime soon.
Hm, I how can we take a 175 LOC project as something relevant in any way?
_This particular_ 175 LOC Java project takes up an enormous amount of memory. For all you know it was just coded poorly and the Go one is a bit more reasonable.
> the JVM is this super-heavyweight thing that's really just inherently inappropriate for a lot of applications.
The JVM introduces overhead but not so much that you can make these sorts of extrapolations.
I'm not going to install the JDK just to verify, but feel free to report back if you get different results.
Not sure what the golang overhead is, by comparison.
1. the JVM itself imposes a high floor (hotspot, many shared libs loaded, ...)
2. the Java language is full of overhead at every level (boxed types are a pet peeve of mine)
3. the Java ecosystem has a tendency to regard memory as an inexhaustible resource, which lead a lot of waste in many 3rd party libraries
Optimization was the original goal, increased maintainability was a pleasant result.
Value types would help so much with this issue. I know they're coming one day. I hope Java/JVM can replicate memory efficiency and cache coherence of C++ std::vector for small objects.
There are enough solutions (e.g. servlet containers) where you can run multiple services in one VM, with isolation, and security hardening (using security manager).
With regards to memory footprint - the difference here is that Java uses a minimum and maximum heap size that can be tuned with parameters. This has downsides (typically more memory use) and upsides (upper bound on the maximum memory use of a process).
Most notably, all services will share the same heap, so one ill-behaved service can bring down all the other services. The only way to prevent this, is to run each service in a separate VM.
And at that point, you are once again comparing one Go runtime per service with one JVM per service.
If you're running a Go app in a container, you can use also tune the upper bound on memory use by restricting the available memory for the container.
The whole Netflix is based on the JVM and Java. They are enormously big, they have high CPU and I/O requirements in many cases, and yet, they manage well on Java.
I'm not a big fan of Java as language (to say the least), but the JVM as a runtime is very-very sophisticated.
You just need the Server JRE which is 57MB tarball + 157MB uncompressed.
I wish they provided the JVM startup info and stats to compare. Maybe JVM's resource usage could be shrunk, too.
The number of LOC has no bearing on how much memory something will consume. For all you know the service could just be poorly implemented.
And not sure what JIT space is.
Still, that's time and effort, and in Go you get all of those things for free. There's far fewer janky edges (I spent hours figuring out the problems with signed jars, shaded jars, and symlinks when trying to deploy)
The other servers that went through a rewrite also ended up being significantly smaller in go but that's a story for another day.
I suspect the same (or greater) linecount gains could have been gotten by using a language even more pithy. How much you wanna bet the equivalent Clojure or Scala versio would be half again as many lines?
263 lines?
Clojure is quite terse, and Scala can be when you're using the right toolsets.
But just because you wrote just 200 LOC, that doesn't free you from bugs and regressions in the remaning thousands of lines of code you rely on.
As an anecdote, at a previous work place we had a program that parsed CVS-files for import into an SQL database. This particular program was written by a researcher (meaning: someone with great domain knowledge, but perhaps not a very strong software engineer) in Visual Basic.
The program worked great, but we had to upgrade the server that ran it to newer versions of windows, and at some point needed to recompile the code (I don't thing this was releated to a bugfix, IIRC it was simply a run-time/linking issue).
As it turned out, the code compiled, but the program didn't work -- MS had changed the API for CVS handling (probably fixing a bug). I don't recall exactly what it was, might have been how floating numbers were parsed/handled, possibly an edge case with Norwegian/English localization or something.
Net result, we had a pretty though debugging job on our hands, for a very small program...
(I feel a bit bad for singling out MS for this -- as I understand it, they are generally very good about maintaining backwards compatibility -- even to the extent that that becomes a problem. I guess we just hit on a corner case with our use-case and this particular old VB code).
Both examples mentioned in the article (the 175LoC program and the CA) sound like very simple programs. E.g. I once wrote a C program which watched some directories with inotify and compressed new files using zlib. The memory footprint was 350KB. Obviously an empty JVM alone would use 100x more RAM. This static ~30MB overhead might be important in some cases and not relevant at all in others. The incremental (per-object) overhead is probably more relevant to almost all real-world usecases which are a bit more complex then the ones mentioned above.
Also - care should be taken not to compare apples (no TM) to oranges: e.g. if you use a huge ORM in Java (which among other things also caches results of each query) and then do a simple SQL query in the new implementation, would be strange to expect those two to perform similar. This happens quite often with rewrites - they almost always aim for simplicity, do short cuts, get rid of "unused stuff". Basically a rewrite from Java to Java will also usually improve performance.
Must-read about rewrites: http://www.joelonsoftware.com/articles/fog0000000069.html . Though, I guess, everybody has already read it.
However, we've been eager to try Go in several places. Believe it or not, what has held us back is the lack of a solid LDAP library. We could/should scratch our own itch and be done with it, we lack the time... still so many things to do! In the mean time, for us, Java support for LDAP is nothing short of stellar; and has been for years.
Can you share what the microservices are doing? What are they for?
- team-server probe: already mentioned in the blog post.
Determines if any installed Team Server is down.
- ca: as mentioned in the blog post, as simple Certificate Authority
- charlie: a checkin service. Desktop clients periodically post to it
to signify they are up. This data is used in each user's device list
to show if the device is up and which ip it was last seen from.
- auditor: takes audit event in an HTTP endpoint and forwards
them to a raw TCP connection as expected by splunk and co
- valkyrie: a relay server used for data transfers when desktop
clients cannot establish direct TCP connection (more about that
in a future blog post)
- lipwig: a messaging/pubsub server used for peer discovery
and notifications (more about that in a future blog post)It's stems from the following quotes; knowing this information, why do AeroFS still think Docker is a good fit for their use-case?
> However, after our move to Docker, we noticed a sharp increase of the appliance's memory footprint.
> ...
> We identified several major factors behind these symptoms:
> 1. an increase in the number of running JVMs, as each tomcat servlet was placed into a separate container
> 2. reduced opportunity for the many JVMs to share read-only memory: the JVM itself, all the shared libraries it depends on, and of course the many JARs used by multiple services
> 3. memory isolation could in some cases confuse some sizing heuristics, which lead to larger caches being allocated by some services
a) Their service was already written in it b) Their service was already performing adequately before they brought docker into play
But those aren't answers to your question. Java may have been a terrible fit for them, but their decision to move away from it was not motivated by that -- it was motivated by their already-written, already-performing Java code no longer performing as well, because of Docker.
The only question is, does Docker bring them enough benefits to justify reimplementing large amounts code, in a new and unfamiliar language? Maybe it does, but somehow I doubt it.
To me it seems the Java solution was wildly over engineered and probably could have been refactored to not even need Tomcat.
"Everything was fine until we switched to Docker" makes me think the trouble may not be all Java. Anyone have educated thoughts on this?
Is JVM overhead shared when multiple Java apps are being run on the same machine?
(Note that the huge size of the classes is also the big reason why JVM startup time is so crap; another reason that multitenant JVM systems are great is that every process after the first starts much faster)
(Edit: not meant to be a political statement, more of a practical observation. I only recently started using Go.)
It's not like golang doesn't have some overhead of its' own... There's also Rust, and D to consider.
http://www.iron.io/blog/2013/03/how-we-went-from-30-servers-...
Regardless, Go certainly improves on the reliability of applications vs C++ while being much more efficient than .NET or Java.
Here's the site: http://www.h4labs.com/dev/ios/swift.html
Here's the data: https://github.com/melling/SwiftResources/blob/master/swift_...
I'm approaching 1400 URL's and it's still snappy. I was hoping Go's claimed "efficiency" would keep freely hosted a bit longer than Python.
That is exactly why I picked my Go for my latest side project.
I just hope the framework I'm using isn't the bottleneck, because I really hate writing pure Go servers.
How does Go stack up for cross-platform development? Does every application and library have to be (re)compiled for the target platform?
What about support for alternative architectures (ARM, PowerPC, etc)?
This isn't so much Java vs. Go as it is JIT/interpreted vs. AOT-compiled. The numbers are entirely typical across a wide range of such comparisons.
With an interpreter or JIT, you need to load all your code at startup and process it. Generally, _all_ of your dependencies need to be loaded and parsed upfront, either converted to some internal in-memory representation or JIT'd directly to machine code. This will allocate a bunch of heap structures during the processing, and the end result is a bunch of data that needs to stay resident and can't easily be shared with other processes.
With AOT, you mmap() in a file. The OS only pages in the code you execute, and can page it back out as needed. Pages are shared between all processes running the same executable.
At sandstorm.io our rule of thumb is that an app written in Node, Ruby, Python, PHP, etc. will take 100MB of RAM while an app written in C++, Rust, or Go will take 2MB. Since Sandstorm runs per-user (and even per-document) app instances, this is a pretty big deal.
The good news is that https://github.com/google/snappy-start should fix this problem: by checkpointing the process after it finishes its parsing/JITing but before it starts handling requests, we can get an mmap-able starting state that is very much like an AOT-compiled binary. At least, in theory -- there's still a bunch of work to do for this to actually work in practice.
> The resulting docker image shrunk from 668MB to 4.3MB
While I would expect the Go image to be smaller (since Go builds static binaries, so literally all you need in the image is the binary), I suspect that the 668MB Java image was at least 90% unnecessary garbage that was not actually needed at runtime. Unfortunately the package managers we all use are not optimized for containers; instead they evolved targeting systems with dedicated disks that can easily absorb gigabytes of bloat.
In Debian, for example, every package implicitly depends on coreutils. That's perfectly reasonable when installing an OS on a machine: you almost certainly need a working shell to boot and administer your machine. But a container can get by just fine without coreutils, and a typical web server probably (hopefully) doesn't need to call out to a shell. Even if a shell is needed, busybox/toybox is probably sufficient and will take a lot less space.
Packages also often contain things like documentation, unit tests, etc. which obviously aren't needed in a container.
For Sandstorm.io we deal with this problem by running the app in a mode where we trace all the files it actually uses, and then we build a package containing only those. It mostly works and manages to keep packages reasonably-sized, but it does lead to bugs of the form: "I forgot to test this feature while tracing, so the assets it requires didn't make it into the package." We're looking for better options.
We need to run the app up until the point when it diverges -- i.e. when it first observes input that will be different across different runs of the app. For that, we need to be watching the syscalls and evaluating each one for potential divergence. As long as we are doing that, we might as well at the same record a log of those syscalls which we can replay later. Then once a divergent syscall happens, we dump the state of memory. Later, we can restore the memory and replay the syscalls to reproduce an identical starting process.
CRIU has no concept of divergence. CRIU takes an already-running process with arbitrary state and snapshots it whole.
CRIU's problem is actually orders of magnitude more complicated than snappy-start's: it needs to understand every possible file descriptor type that the process could have open, every aspect of process state, etc. snappy-start only needs to understand the specific syscalls that we care to implement; it can simply consider any call it doesn't recognize as divergent, and stop there. Adding support for more syscalls is then merely an optimization.
CRIU also requires special kernel features to support, which means more attack surface. Sandstorm wants to block everything except the most common kernel APIs for security reasons. snappy-start requires no new kernel features; it uses the well-understood APIs debuggers use, and we know we can still prohibit apps themselves from using those APIs.
Meanwhile, CRIU is much harder to customize. How would we decide when to do the snapshot? We'd have to re-implement much of snappy-start just for that purpose. And how do we teach CRIU about the specific assumptions that are safe and useful to make given our particular environment?
None of this is to say that CRIU is bad -- it's actually pretty amazing. But it's not the best fit for this specific problem.
> I suspect that the 668MB Java image was at least 90% unnecessary garbage that was not actually needed at runtime. Unfortunately the package managers we all use are not optimized for containers Exactly. That was part of the point of this blog post, which I may not have been successful at getting across. Switching to go was, if not the path of least resistance to solve this issue, at least one of a few relatively easy routes. It also happened to be a great deal of fun.
We're planning to open source our build script shortly.
Isn't the JVM + servlet container thing supposed to be able to isolate the different services? They get their own bunch of threads, and their faults probably shouldn't crash other servlets?
Sun was really ahead of the game in various fronts.
If Sun had more tightly controlled the marketplace for these things I think the whole ecosystem would have been more robust.
Now it is more common to run Tomcat or Jetty embedded for your app so that they are isolated.
It is absolutely not a community project. Go governance is a 100% @ Google . There is no go committee or go working group outside Google. It's backed by one company that has full control over it. Sure it's open-source, but good luck with a fork. How can anybody be so misleading about that fact ? What did make you think Go is a "community project" ?
> but that does not mean it depends on Google.
There is a top down , vertical relationship between the Go team at Google and the rest of Go users, Go main goal is to fulfill the Go team needs, period. If you find it useful then it's a bonus. That's exactly how the Go team speaks and act, in fact the Go team makes it really hard to contribute to the core.
Please, stop saying what you say, it is completely untrue and a total mis-characterization of how the Go project work.
I still want to know what made you think Go is a "community project".
I know of one fork that will provide some lucky grad students with a degree. Can't find the link right now.
Well, the main developers do listen to the community.
Any project can fork, if there are enough people who are very dissatisfied with the current management.
I'm not aware of any serious grumblings about such a fork though. Why?
The core team is very good, and very narrowly focused. They've communicated clearly on nearly every issue as to what they're doing, and why. It seems clear that they are intent on making the best possible tool that fits with their particular vision. There are people who agree with this vision, and broadly support their efforts. And then there are people who really don't like their vision, and wander off to use Rust or something else.
The core team has enormous respect from the existing golang community. If Google suddenly laid off the developers (or just switched them to something else) or otherwise dramatically changed direction in their support for the project, the community would move in quickly to help out the situation.
I could see people getting together to form a non-profit foundation that could at least pay for a few guys to continue to work on Go full-time. But currently, there is no apparent need, so it hasn't been done. Google is willing to pay substantially for the development, and I don't see gophers complaining about that.
It's not going to happen, for the same reason the biggest startups in the Silicon Valley never came together to write their own language before. Let's get things straight, the only people who control Go is the Go team period. You are not talking about facts, so i'm not sure what is the point of your comment.
This kind of thing has happened in the past: XFree86, OpenOffice, gcc, etc.
Question #1: How invested should you model Google as being in Go's future? Answer: Extraordinarily invested. They have a whole lot of code written in it, most of which the world will never hear about, and which powers services that they will continue running until the sun goes nova. Google will likely continue maintaining and extending Go programs (as well as C++, Java, and Python) for the foreseeable future.
Question #2: Can Go survive without Google's corporate backing? Answer: Go is an OSS project. Like many OSS projects, it receives a substantial amount of support from corporate interests. Go is not uniquely a Google priority -- many organizations like having a systems programming language which has its feature set. In the absence of meaningful support from Google, Go code existing as of August 2015 would continue to function. Further versions of the language/toolchain/etc would have substantial question marks about them, but the community feels large enough that this would be a Significant Event rather than a death knell.
Does that help?
Would I be better off maintaining decade old COBOL projects? Sometimes I think so. Other times I'm glad I can chose the technology I want to solve the problems at work.
August 2015. 31 years since my first computer.
For a while I though Haskell was the most awesome thing. Good Haskell code has a timeless quality to it (in a couple senses of the word). But I found adapting my thinking to it difficult, and it wasn't likely to be something I'd use at work then or now.
(I'm not too familiar with Nim's community so I can't speak to it.)
For the record, I've chosen to use Go over alternatives, although Nim sings to my very soul.
1) Go does not have a runtime. This means that there is no JIT to do any optimizations based on runtime profiling.
2) Go is also designed to compile fast. Since it compiles fast, the compiler's time budget to do compile-time optimizations is small and it probably can't do the best job possible.
Are these two points accurate? If so, how does Go perform as well as it is claimed to do? Where is the catch?
2) Go compiles fast because it gives up a lot of modern features (namely generics).
While it is absolutely true Go servers use less memory that classic servlet deployments, The question is what were you using at first place? where you using a big framework? with this or that big IoC container ? with a bloated ORM ? ... or where you using barebone jdbc and writing servlets without any framework? because essentially that's what you're doing with Go, Go has 0 big framework(and no Beego or Revel are not complex), 0 big ORM , 0 IoC container ... And while the Java culture is about heavy decoupling and reusability , the Go code culture is basically about writing correct code without thinking about re-usability at all, because often you just can't. So I'm curious. My point is didn't you reduce memory usage because Go forced you to write code a certain way ?
That's a very strong statement. Do you have evidence to support this? I don't code in Go right now, but have been keeping an eye on it as it continues to gather momentum.
[0] e.g., Haskell typeclasses
[1] e.g., Java
If your library works with streams of bytes, its methods had better use io.Reader and io.Writer. If you're writing an ORM, it had better be built on database/sql.
> the Go code culture is basically about writing correct code without thinking about USABILITY at all
The reality is go is YOUNG and it shows. Lets take two good examples that are part of what is going on in golang today.
Vendoring: The core teams approach, and what the community are doing have diverged pretty rapidly. Recently there have been some efforts bring vendoring into alignment, but what is being used isn't the best approach/solution, its good but on the whole could be better.
http: The core library is great if you want something small, and its fairly easy to put together a tool chain that will remain "idiomatic". However if your going to go out and build something "large" then you have an interesting issue, because OOTB core http lacks any concept of context. There are some interesting tools and frameworks out there to make up for this (gorilla/context, codegangsta/negroni) but if you adopt them your no longer "idiomatic"... your libraries are now less reusable because they are tightly coupled. It looks like a new method signature is going to be required in the http package, one that uses golang.org/x/net/context and returns errors so we can have a sane http stack OOTB in go....
Log vs syslog: Logging in go is fairly messy right now... there are a bunch of packages that try to make up for this, but really better logging has to be built into the core. Syslog has levels/features, log is just dry and basic (maybe too much so). The core not only needs a unified solution, but one that is going to be context aware.
Will these get fixed? Probably! The core team isn't deaf. However if they don't start moving on a path to 2.0 soon and address some of the real issues, I fear that go will end up in the same boat that python did in 2.x vs 3.x.
P.S. in spite of all this I have been writing a LOT of go lately and enjoying it, but it really does need to grow and soon!
This is why there's such a strong pressure to move to more svelte language implementations and "microservices" as you start to see containerization take off.
What a ridiculous statement. Go encourages elegant and simple interfaces to make code reuse easier. The tooling provides an incredibly easy way of sharing code. The "culture" is all about sharing code. Go's much better at facilitating code re-use than (say) Java, both at a language level and at a cultural level.
The JVM has the largest library collection of any platform (currently 1,026,516 libraries just in Maven Central repository). Sure a lot of is to do with the age of the platform but this idea that Java is somehow impeding code reuse due to language/cultural reasons is ridiculous.
One of the biggest complaints for people coming to Java is actually just how many transitive dependencies libraries are brought in.
Further, if I wanted maximum dynamism and runtime hackability I'd go for a Smalltalk.