Python at Netflix
techblog.netflix.com
techblog.netflix.com
There was many interesting things listed at the blog post. I have been planning a system similar in idea to Sting, your OLAP server (?).
I'm planning a light RPC style system on top of Pandas dataframe objects, transferring HTTP requests to Pandas method calls. Memory usage of such a system with our data is too expensive if all data is in use. So, I'm investigating possible caching and database solutions.
Could you perhaps elaborate on the implementation of Sting's in memory database?
[posted from yet another HN account]
So now that's +2. :)
The idea of marrying "slow" Python and OLAP is disturbing my sleep!
P.S. I really enjoyed your talk at QCon SF last year!
Ruby and Python are similar in terms of execution speed and developer productivity. If your work requires something that Python has and Ruby doesn't(say numpy), then it might make sense; otherwise what would you accomplish by switching from Ruby to Python?
Simpler syntax?
Posts like these are great on a tech level, but they're also good recruitment - Seeing Python, along with hackers that I respect (like you) at Netflix make me at least the littlest bit tempted to go there ;)
But in all seriousness, we like to share what we do because sharing is cool and it attracts other people who like to share cool technology, which are the kind of people we like to work with.
Wakari (http://wakari.io) is nearing 1.0 and we're about to turn on paid monthly plans for beefier nodes and GPU nodes, and we have also been in talks with several companies about in-house private deployments.
Give us a shout or shoot me an email at pwang (at) continuum.io! :-)
b) python does have first class functions?
c) threading still happens, you just can't take advantage of multiple cores. This isn't as big of a problem as you'd think though, you can still write very fast code.
Also, packaging is a whole different can of worms (but hopefully will be sorted by distutils2!).
Python lets you pass functions other ways than using lambdas, you know. If you've defined a function with the standard "def foo" style, you can pass "foo" as an argument.
Even the standard "1-line lambdas LOL" limitation can be circumvented by defining the function elsewhere and making the lambda a call to it.
I have been building large projects in Python for the last 10 yeas and never really thought "Hmm I wish I had more powerful lambda functions". You can always just have regular function and pass that around. Why do you need lambdas so much?
First class functions simply mean that a language treats functions as it would nearly any other data type, namely that they may be passed as parameters, bound to variables, and returned from other functions.
Python's poor handling of lambda expressions do not invalidate that functions are first-class citizens.
Java shines at the huge end, but for most everything else python is a more productive choice as it is so much more concise. Perhaps you are not yet using it optimally?
Hit me up on email (it's in my profile).
I've also pretty much replaced bash with the ipython shell at this point, as you can use all of the bash commands anyway and even seamlessly mix them with python, file_list = !ls, it's a excellent environment for data analysis and manipulation.
While I've got you, and seeing you are not longer at Reddit, what do you think of jkl.io (launched today)?
Put another way - when you think of python for web applications / services - do you think rapid development and speed, or just rapid development and you would go somewhere else for speed?
The whole web application is not CPU bound. The parts which are CPU intensive are rewritten in performant language. For eg, if you are twitter, you still render templates in Rails, and write CPU bound services in Java/Scala. Replacing the whole app with Scala/Java will be pointless.
That doesn't mean that every time you need a service, you can't use Ruby/Python. If all your API server does is check cache for data, load data if there is a cache miss, encode json and return it, Python/Ruby will do just fine.
At reddit we solved the problem by writing the most often called parts in C and using c extensions. There is definitely more work to do there, but overall I think Python gets a bad rap as far as speed is concerned.
It's certainly not the quickest, but it isn't a dog either.
I'm wondering if anyone has solved these sorts of problems with Cython in the real world. It looks like a great solution for a large set of problems in terms of Python performance, since it becomes a CPython extension, but I never hear anyone talk about it. Can anyone chime in?
The entire scientific Python ecosystem has basically moved on to Cython. Its use is quite prevalent, and it's basically displacing SWIG as the preferred wrapping tool of choice.
Why? The apps are mostly IO bound, and any large site is going to be caching, deferring slow operations, and sharding/breaking into tiers, cdns, etc, etc. The amount of time a web framework actually runs is relatively small.
Others have already mentioned what to do when performance is a problem.
Maybe that is why you were down voted. (FWIW, I did not down vote you.)
Given our culture, there really is nothing stopping anyone from using it, other than the fact that they would have to figure out how to interact with the platform.
But overall go is a really neat "systems" level language.