Comparing a web service written in Python and Go [pdf]
indico.cern.ch
indico.cern.ch
Of course something compiled directly is going to be a bit faster, but development time is important too. Python has more libraries and is (for many people) probably faster to write.
Serving multiple requests is best utilized using a preforking webserver in front of Python, whether Apache, nginx, etc. This allows multiple requests in without any async voodoo code. Twisted for example is not the right answer in this case, because it doesn't get you multiple processes and messes up the way you write code (async event driven code is more time consuming to write/debug).
On the backend, your webserver does not start longrunning backend processes, but you can launch them using things like celery, which is a process manager that allows you to start jobs and so forth. Celery can run on any number of machines, and your backend can scale independently of your frontend if you wish.
Historically, some very computational parts of Python were often written with C bindings. While I haven't done so, things like Cython may also be promising for extensions. There's also things like ctypes for quickly just taking advantage of native libraries in a Python function.
Personally, given, I like how Go has things like channels, but I would never adopt a programming language for just one specific feature when I lose out on other features that are valuable to me, for instance, an object model.
(I'm also really curious to see how the typing options in Python 3 play out)
Anyway, I mostly wanted to point out as most people are doing web services that you should be fronting Python with some sort of web server that allows preforking, and then the concurrency issue, in my experience, becomes not a thing.
Many backend libraries can easily take advantage of libs like microprocessing, which are not the most 100% friendly in their more complex IPC-type cases, but are pretty workable.
That really depends on your requirements. If you need multithreading (not multiprocessing), you cannot use Python.
About to show my ignorance, but when is multithreading useful when multiprocessing is not? Assuming it is a use case that is suited for a high level dynamic language to begin with.
That kind of sounds like you're nitpicking over an implementation detail. Which brings me back to my original confusion. If I have an interface that allows me to articulate concurrent tasks, I don't see the difference between threads, processes, or separate machines.
Again, this is assuming we're not in a resource constrained environment, and we're not doing something so processor heavy that it really should be compiled.
Python threads are real threads, and things like blocking socket IO does not block Python execution in other threads.
This only matters for performance-critical applications.
If you need multithreading of Python code for parallelism, you cannot use CPython (and, consequently, can't use Python 3.)
Both IronPython and Jython use native threads without a GIL.
I absolutely agree, but I also think that deploying python on the web still has too much of a learning curve. Even the standard nginx > gunicorn > wsgi model is kind of a pain. Couple that with celery, and init systems, and you're basically down a sysadmin rabbit hole.
Go has an object model.
I'm no sure go creators would agree with this. The go object model is different enough from other object models that it would not qualify as one for a lot of people.
Spot on. Concurrency vs parallelism, and clean distinction of responsibility (web server vs backend threads).
> Many backend libraries can easily take advantage of libs like microprocessing, which are not the most 100% friendly in their more complex IPC-type cases, but are pretty workable.
This is painful. In Javascript I can be careless (well to a great extend for people like me likes magic) using promises. Python can achieve this too but with a great effort of learning either coroutines, gevents, or asyncio. Though I have to admit that Javascript has its own problem facing parallelism.
I have done things with gevents, spawning greenlets and respond to user immediately. The thing is, backend should always be stateless, so worker models like celery and rabbitmq pub/sub and etc are more popular.
Go is a good tool for the job, Python threads are not. asyncio or one of the event-based IO frameworks should work much better.
As for the problem of sharing data between processes (slide 5): it appears that this service is read only? If that's true, what do you need to share? Every process can have it's own connection pool. You don't even need multiprocessing, just use SO_REUSEPORT and start your application multiple times.
[Edit] I'm a assuming a CPU with 8 cores, not some 64 core monster.
I imagine that context switching between 175 OS threads all in the same process wouldn't really be that big of a deal.https://www.quora.com/How-does-thread-switching-differ-from-...
Additionally there are many legitimate cases for for a lot of threads like disk IO. If you find your self having to push a lot of bytes to/from a high iops drive like an SSD / NVM drive. Unless you're doing large sequential transfers that you can do in one large call, you will needs submit many concurrent request to saturate the drive (via threads). Disk IO is not network IO.
Maybe you are right and it's one of those urban myths that we sometimes carry over from times long past based on assumptions that are no longer true.
I would love to have more hard information on that one, because I think that the currently fashionable async/event based way of doing a lot of things makes programs much harder to understand and write.
In python, it is at lease 10x less code for json parsing. json.dumps(), json.loads() is basically what I needed. The exception handle fill in all the undefined easily.
Also, go used a lot more memory because the GC is not under my control. In python, I can tell the GC to collect and one can see the memory shrink immediately. In go, that was not the case for me. A program that build GB of search index database in go end up using 4x the amount of memory as compare to python. Golang at that time (2+ years ago) lack the gc debugging infrastructure for me to resolve the problem.
What makes Java code look bloated is a culture of overdesign, not so much the language itself.
Best opensource the code for both and the benchmarks and have people go at it.
Quite frankly, I'm getting a little sick of the arguments that happen on HN whenever a Go-related article comes up (and particularly so with Python vs Go). You get people who seem to hate Go who go all out to criticise Go and/or statically typed languages. Who argue that pro-Go articles are biased; and so forth. And then you get the Go fanboys (of which seem to be less vocal lately) who declare that Go is our lord and saviour and we should be rewriting core OS internals in it. It's just nuts. Sensible people would see that Go is better at some things and worse at others. And that articles like this are just an interesting case study and might not apply to their own personal software problems but not in any way biased beyond the fact that the article is tailored specifically to their own software problems.
All these languages are statically typed and in my understanding of Programming Languages, way better than Go.
Nim's often interested me. I'd really love to try it but sadly it's lack of mainstream support means it would be hard to justify using Nim in any professional capacity, which vastly limits it's usefulness to me. And sadly I don't have enough free time to learn languages for fun these days.
Unfortunately this overview is light on any meaningful details. As a general rule a rewrite of any project will result in fewer lines of code, however, in general, a rewrite of any project is a terrible idea.
Given that this seems to be a situation in which you have a lot of blocking waiting for concurrent requests, why not try something like gevent?
It's good for people to try different approaches and technologies. I'm glad they managed to have success with Go, that's good for everyone. It would have been interesting for the reader to see some of the gory details of hacking around with the existing codebase to see some of the ideas that may (not) have worked.
On that note, I would suggest that if you think they could see big gains with little code refactoring simply by switching Python frameworks or even to a different Python runtime, then maybe you should contact them. I'm sure the author would be open to ways to increase their throughput with less developer overhead.
Feel free to rewrite their old Python app in Python for free. They may thank you and even use your port.
Java does not have type inference, does it? That alone makes it more painful than other static languages.
Does Java have type inference?
Even still, it's only a bit of typing, which has never really bothered me. Once typed, the available refactoring tools make it super simple to edit, and I find that I spend way more time editing code than creating it from whole cloth.
* the stuff you want to write quickly but don't mind it running slowly, i.e. throwaway code
* the small stuff you know won't become a larger system one day, e.g. 30-line Gradle build scripts and code testing Java classes
* when you know you won't be upgrading to Java 8, which Groovy hasn't kept up with syntactically
Groovy's static compilation was tacked on for version 2.0 and doesn't work, except for sprinkling the occasional @CompileStatic around your code in a trial and error fashion. You didn't actually say you HAD started a new statically typed project in Groovy on the Java platform. If you do, use Java 8 which makes much of what Groovy brought to Java 7 redundant, or another language written from the ground up for static compilation, e.g. Scala or Kotlin.
In fact, I've found even for code testing Java classes, using Clojure is more productive than Groovy once you get over the syntax hurdle because macros can eliminate verbosity in repetitious test scripts in a way functions can't.
So it was easy to switch to Python and come to the conclusion that it was all the static typing's fault, because that was the most obvious difference between the languages of the time and Python/Perl/etc.
But it's 20 years later now, and the statically-typed languages have not been standing still. They're a lot easier to work in now, even for prototyping in my experience. Go, which I mention since it's on topic, is reasonably fluid with its interfaces and structural typing. It isn't quite Python's level of fluid, but I also find it doesn't take that long before the static typing wins start to balance out anyhow when it tells me about problems before they've metastasized because I didn't know about them and kept going.
The whole functional programming world with its type inference is fairly fluid to work in, and when it gives errors, I find they tend to be real errors, even in your prototyping code, that really do need to be addressed before you can compile. (If you get good at it, prototyping is actually really easy in Haskell, but there is an up-front investment.) Optional/incremental typing, despite my general distaste for them [1], also can be easy to work with. And even some of the old school statically-typed languages are easier than they used to be, because you can find some abstraction layer that isolates you from the sort of thing that made the 1990s a really crazy time to program in.
So, it helps to use a language that doesn't date back to the time of Python's origination.
The only thing inconvenience that truly static typing brings to prototype code in my experience is that it really wants you to push a given change all the way through the code base, when you may just want to experiment with it on one code path. Generally there's a way to just beat the compiler down, though. And when you've learned the art of using types to mean things, and when you've learned to read what type errors mean and that they are not just a content-free shouting of "NO!", the ability to harness what the errors mean and react to them quickly in your design, rather than bake them into your code accidentally, can end up being a net gain, even in mere hours of work.
[1]: Mostly a consequence of the fact that, as I'm describing in the rest of this post, static typing is a lot easier than it used to be, which means I consider incremental typing to mostly be solving a problem that no longer exists. YMMV.
What type system feature does Golang have that was invented post 1990?
Interfaces certainly were not created post 1990—Java had them, to name one obvious example—and structural typing is very old as well (and I don't think the sort of pseudo-structural typing that Go has for interfaces is particularly important for usability anyway).
Popularity and a set of libraries useful enough to do real work in it unlike anything else I know with structural typing.
What feature does Rust have that wasn't invented in some academic paper by the 1990s if not sooner?
An advance doesn't much matter to people doing real work if it doesn't come coupled to a practical language and a set of libraries that enables real work. This is sad in a way, because "the set of libraries that enables real work" bar is going up every year, which makes new languages harder. See also "Why did Haskell need 15 years to take off?", the answer being "that's when people started putting together practical libraries" among other things.
In 2025, when $PRACTICAL_DEPENDENTLY_TYPED_LANGUAGE is finally taking the Hacker News world by storm, I will not take kindly to suggestions that we could have and should have been using it in 2015 or earlier. No, we couldn't and can't. They're not useful yet, no matter how theoretically beautiful they could be.
> What feature does Rust have that wasn't invented in some academic paper by the 1990s if not sooner?
Actually, the borrow check is novel (though you can find precursors in Cyclone and fractional permissions work), and I think that is responsible for a lot of Rust's popularity (as it enables memory safety with dynamic allocation and no garbage collection, something that is actually new). But the answer you were looking for is "nothing", and that's precisely my point in regards to Golang. This stuff isn't new. Specific repackagings of it are getting popular, but I think that factors unrelated to the technical details of the type system have been largely responsible for that.
(I also continue to disagree that structural typing as implemented in Go is a useful feature—I think it's a net negative for usability because it saves a few keystrokes at the cost of precluding implementing interfaces for types outside of the package they were defined in without writing annoying wrapper structs—but that's neither here nor there.)
It's a very pragmatic answer. As someone who generally straddles the pragmatic/theoretical divide, I mean that both fully as praise and fully as condemnation. ("Do I contradict myself? Very well, then I contradict myself, I am large, I contain multitudes." - Walt Whitman)
I never did hear what Rust decided to do about what Haskell called the "orphan instances" problem, I only heard about the problem... what happened there? (Honest question, not critical in any way.)
>[Go is] way to easy to program than Erlang
DAS = data aggregation service