Open Sourcing Our Go Libraries (2014)

Your comment reads to me as if you think the author implied that they think Go "magically" provides scalability. I inferred nothing of the sort. Rather, I assumed they wanted to use a language which provides concurrency as a primitive, to make it easier for them to write concurrent code.

This switch (Python to Go) for this motivation (better concurrency support) seems reasonable to me, particularly since Python does not have a good concurrency story. (I love Python. But I would not choose it if I wanted high concurrency.)

Everything you said is true, of course, but I think it applies more generally to inexperienced startups with first-time technical founders, or larger corporations with a "B" technical staff.

In this case, you're talking about Dropbox, which already has achieved the scale, in both the large and small senses, that require a deeper level of problem solving than simply switching languages would provide.

Even if I personally find Go an odd choice for this project (why not C++, which already has the library support they need, and much more?), their present-day successes demonstrate that they didn't expect a scalability panacea from Go, just a better runtime for a subset of their critical path code.

Nothing is magic but in this case moving from a language with a GIL to a language built for (single machine) concurrency probably scales a lot.

The OP says as much in the comments: "For us, one of the biggest latency wins comes from the fact that go can truly execute sql statements in parallel (whereas python's GIL serialized these parallelizable operations). In general, single-threaded go is at least 5x faster than pure python (without c-module)."

Go over Python may not be a magic bullet, but it's a damn useful tool nevertheless. While Go does not magically scale to datacenter-level systems, python cannot, without significant work, use even the resources on a laptop. There's a huge range of problems that are larger than 1 core and smaller than 48 cores (or 72 cores, or however many are in your largest server). And as the sibling comments have mentioned, straight-line single-thread performance is not irrelevant. Starting with a language that's slower than perl isn't a good beginning.

You're right regarding larger system architectures where it's more about the services and less about what language each is written in.

1) Regarding concurrency, Go really pushes you towards writing things in a scalable way with the goroutines and channels, while still giving you mutexes for when those are the best fit.

2) Regarding single-threaded performance, this does indeed start mattering once you un-bottleneck-yourself on the architecture and concurrency fronts. 250 boxes are cheaper than 500.

You're right, but consider this: they managed to rid themselves of Python's GIL by not using Python. If they can actually reap the benefits of parallelism, it means a lot for scalability.

It's a slightly odd comment, SQL queries performed by a C driver are one of the times that you actually get to release the GIL and let the thread do its work without the python interpreter. Depending on the SQL engine, you're ok waiting for the sql backend in parallel, but you may suffer single threaded python object creation processing the result set.

They are not executed in parallel though, only asynchronously.

that is interesting - does Gevent not give this in python (or twisted) ?

The first thing I usually do in a new language is find out how people use it. With go and go fmt, I don't have to worry about most of it. (golint and go vet cover my other mistakes most of the time)

Did they just run fmt on it?

The first thing I usually do in a new language is find out how people use it. With go and go fmt, I don't have to worry about most of it. (golint and go vet cover my other mistakes most of the time)

The first thing I usually do in a new language is find out how people use it. With go and go fmt, I don't have to worry about most of it. (golint and go vet cover my other mistakes most of the time)

Groupcache addresses a very narrow use-case of caching: immutable data. Key's in groupcache cannot be changed or deleted, which is what allows for some of the cool things it does like distributing keys to multiple nodes automatically and prevention of stampeding. It's useful for things like caching lots of small static files (which I believe is what google uses it for), but it's not useful as a db cache where things are constantly changing.

Just glancing at dbox's caching package it looks like a much more general cache, with deletes and sets and all of that. So the two aren't really comparable.

Probably because of other services--not necessarily written in Go--which already use memcache. Don't fix things that aren't broken, right?

1. Go, being a statically compiled language/having a closer to the metal memory model, generally has higher single threaded performance.

2. Go allows true concurrency in many situations Python cannot (due to GIL etc); Go also supports lightweight threads that are multiplexed over actual OS threads.

3. Go makes it easier to control and reason about heap allocations.

4. Go is even easier than Python to integrate with C/ASM code.

Since they mention "scalability", I think they're more concerned with the fact that Go has concurrency baked into the language. Python does not.

It's a statically-typed and compiled language, by most accounts you can expect it to be faster in a lot of places.

This has more to do with the interpreter, of course, you can compete with PyPy.

Go has a much better runtime system than Python (can spawn lots of green threads in parallel using goroutines), but it's also broken in so many ways (e.g.: type system).

We still have tons of Python code, and likely always will. Therefore, Pyston is still really interesting to us.

Only specific, common, core services (storage etc.), are being written in golang; the "long tail" of application code on the backend is (and will remain) in Python.

- Dropbox infrastructure engineer

Seems to still be actively developed: https://github.com/dropbox/pyston/commits/master

Snark aside, I don't see what one has to do with the other. It's perfectly sensible for them to keep pushing pyston to have an alternate solution.

I expect your detection is a little broken :)

Much more along the lines of:

* Start with Python

* Have a great time and build successful apps with it

* Become global-scale infrastructure provider

* Hit problems that only occur at that scale and switch tools

The take-away is, you're almost certainly not going to end up at the size of Dropbox. If you're spending ages as a startup futzing around using bleeding-edge technology just because it's cool, you might well be wasting time – or at least prematurely optimising.

This is the perfect example – Dropbox has the engineering talent to write a set of libraries to implement basic features. Earlier-stage companies don't have that luxury.

Dropbox is not dropping Python: "Dropbox will continue to develop majority of its features in Python. We have only migrated performance critical components to Go."

I'm detecting a different trend:

> HN post about any language > Snarky comment about any language > Language war

It's depressing that your comment has percolated to the top of the thread; it has nothing whatsoever to do with the story.

I don't think they found out Python is horrible - the first line is praising the language, it just didn't suit their needs anymore.

Except they've managed pretty well with Python for everything for several years. And we're talking about Dropbox here, not some small startup. Maybe it's not "a little horrible".

Besides that, they're migrating only some parts of the backend. They're not "switching". The rest is still Python.