Web Service Efficiency at Instagram with Python | Hacker News Reader

Web Service Efficiency at Instagram with Python

engineering.instagram.com

195 pointsksashikumar51 comments

Instagram is a great story for Django, it's the best argument to bring against doubters of Django's scalability. But I wonder how much Instagram contributes to Django itself. I follow Django's development relatively closely, and I'm not aware of any major work contributed by Instagram engineers. They aren't listed on Django's fundraising page, neither.

https://www.djangoproject.com/fundraising/

As far as I know, Disqus (https://disqus.com/) and Eventbrite (https://www.eventbrite.com/) are also major users of Django.

Some of the most prolific Django developers work at Eventbrite. I can't say how much, if any, of their contributions to the Django ecosystem are on Eventbrite's dime, though.

Well... Andrew Godwin ( https://twitter.com/andrewgodwin ) works at Eventbrite. I think he doesn't need any introduction :)

For lazy people: he wrote South, Django migrations and he's currently working on Channels.

The fundraising page is depressing. They're only asking for US$200,000 and they only have 19% at half way through the year. In other news... Instagram hits half a billion monthly active users... but not on Django's fundraising page.

I'm not knowledgeable about open-source issues, pros/cons, etc., but man... only 19% of $200K. Doesn't seem right given Python is one of the major languages and I believe Django is the most popular web framework in Python.

You'd think that companies who profit from open source projects would support them financially. Considering how big django is there should be many other companies who profit from the projekt.

200k/year should be collected in no time...

I've found that the engineers who build products on open source software very rarely have the authority to donate company money, and the businesspeople that do have the authority rarely understand the benefits of it.

I think a guide on how to get your company involved in donating to OSS projects would be beneficial. Has anyone come across any good advice, or have any experience in influencing your employer to donate?

No, but I'd love to contribute to something like this.

I tried to organize some OSS donations at the last big company I worked for. Everybody approved of the idea in theory but the red tape involved in making it happened killed it.

I think the problem in many cases is that the finance and legal people at companies big enough to make a sizable donation are so busy that stuff like this never makes it to the front of their priority queue.

I think an easier strategy might be to work into your contract that if you happen to fix FOSS software on company time (to fix things your client notices), your patches go back upstream.

This, at least, is the implicit policy at the company I work for. It's used in the pitch ("we use and contribute to open source software, which we believe leads to a better internet" and so on) to the clients who appreciate such things.

That said, there might be too much legal tape to get that into a contract.

Funding an open source project you use means funding the competitors.

A is a company, B is an individual with lots of spare time.

B feels that he can help the company by writing a tool for them. He also envisions that the tool could be used by other companies and so wants to open source it so he can provide services around the tool.

B says to A : see, I'm going to develop the tool for you for free, in exchange you test it.

Note the tool is the n-th implementation of common business practices, such as an ERP. The tool is not about something specific to A.

B thinks : I'm independent, I'm not hired by A, I write the code as free software as it pleases me.

A then says : wtf? you're going to publish that code on the web ? Therefore my competitor might get it so, no way. I'd be liable for giving away some value of the company to the competitor

So, instead of saying something like "hey, I use that program so let's fund it a little", A actually says "by helping B to develop the program, I may actually help my competitors, I'm in legal trouble".

B is somewhat screwed :-(

(I let you figure out if this actually happened to me :-))

I don't think that this applies to this very case though. Django was not made for Instagram, it was made available to the general public for free, as an open source framework. Instagram engineers only decided to use what was on the table.

I also believe that Instagram needs Django to be OSS, and the same goes for any other framework/language/library that other companies use, as OSS in itself is an extremely powerful label for a project. It brings together developers to form a community with lots of great minds and inputs. It helps feasible projects become production-ready and stable much faster than if it was closed source as its use-cases are far more wider than what a single company would provide.

Taking this from a competetive standpoint isn't very realistic in my opinion. I dare say that if either Django or Python weren't open source software, Instagram wouldn't be built on it in the first place.

I think it's absurd how Instagram can grow financially fat from the hard work of the Django Software Foundation without even contributing a dime to their fundraising or provide engineers exclusively to the DSF. IMHO Instagram should bring that fundraising up to at least 100%, considering Instagram revenue is estimated to hit $5.8 billion USD by 2020, $160,000 USD per year is a small price for the amount of value Django brings them.

It's a topic of morale though, Instagram does not owe the DSF anything, but I guess this shows where Instagrams appreciation lies.

You're right

>>> It's a topic of morale though, Instagram does not owe the DSF anything, but I guess this shows where Instagrams appreciation lies.

That's the point I was actually making. My experience that when you have people brainwashed with "competitive advantage", "profit maximization", and such, well, morale just doesn't exist anymore. In my case, I was surprised to see how quickly someone I trusted can actually switch to this brainwashed brain...

That's the state of the entire Python community. To get money, you need to get hype, and Python is anti-hype. We are people prioritizing on robustness, clarity, documentation, etc. Proven useful things. But because it disregard what we consider superficial and frivolous, we cut ourself from the same enthousiam and energy you can see in other community like JS or Ruby. And from money.

Look at the PSF budget, it's so small it makes you cry.

My thoughts on this:

* Let's stop talking about 'Donations' and start talking about 'Investment' - the conversation needs to be about the pay-off that businesses will get, not some altruistic "give-back"

* Make the case that collaborative investment has a great cost-reward, as the cost is spread over a bunch of companies.

* Make it clear that ecosystems that are well invested will be more competitive that those that are not. If your company has $$$ worth of engineers experienced in Django, you'd be daft not to ensure that Django remains competitive vs other frameworks.

It's an atypical pitch to make, and as with all investments it's difficult for folks to judge, because the pay-off is at a distance. I still believe it's still possible for us to change how collaboratively funded OSS is seen, once companies start to recognize well-invested ecosystems as a competitive advantage.

And yeah, this is pretty much the pitch I'm making for funding Django REST framework, where I'm currently 1/3 of the way towards being able to focus on it 100% of the time, with a full-time salary... https://fund.django-rest-framework.org/topics/funding/

From the perspective of scraping their website, it was React & GraphQL that I was interfacing with. So what does Instagram use Django for? Perhaps that's why their contributions to the project are low.

From Instagram's medium post

> Instagram currently features the world’s largest deployment of the Django web framework

I think it is fair to assume that they use Django pretty heavily from that.

No need to assume when there's evidence to the contrary. Do as they say, not as they do. :)

This part I'm skeptical:

  "Compared to CPU time, CPU instruction is a better metric,
  as it reports the same numbers regardless of CPU models
  and CPU loads for the same request."

CPU instructions will be more stable than CPU time for sure, and it does show that their metric is stable based on their chart but a single CPU instruction can take multiple cycles, especially if there are stalls in the pipeline or other processes are "polluting" the cache. Depending on the CPU model, the number of uops that can be issued concurrently varies, and the latency of these instructions will also vary on the size, and access pattern to their memory hierarchy.

Also, what about a change in compiler version? That can also vary the number of instructions. Unless they are referring to a Python bytecode instruction as a CPU instruction.

Would measuring CPI be a better indicative of their efficiency? They could also track both, no need to settle for one.

I agree with your skepticism. They will not detect a performance regression when there is a change in instruction mix or cache miss ratio.

Wow, CPU is the limiting factor? I've never been CPU bound in a Web app, only I/O bound. Am I doing something wrong?

Granted, I don't use python anymore for Web dev. I'm using languages that can handle 10x the traffic per box that Django used to be able to handle. This isn't a knick against python. I love python. Just such a different perspective.

This also immediately popped out to me as the wrong assumption to make (and start off with, for that matter)... But I could see it being true if:

- You are essentially maxing out incoming and outgoing network IO

- You have no database to speak to (or your DB is in-memory)

In that case, I guess it is just a race to do some computation and fetch the right data from RAM, then the limiting factors would be RAM access and the number of the computations you could do to try and keep up with inbound requests?

What languages/frameworks are you using now?

I'm guessing that you're working on something that is fractionally as successful as Instagram. Getting to the point where you need to scale is hard, scaling is easy.

If engineering time wasn't an issue, how much would a company of Instagram's scale save by switching from Python to a lower level language like C++? Pretend you could wave a magic wand and overnight have the entire code base transformed from Python to well written C++, how would that affect the hardware requirements, and future development?

I guess what I'm really asking is what pros and cons are there with using dynamic and relatively slow languages compared to faster languages like C++, Java, or C#? My uninformed reaction is that it's easy for startups to get started with these easy languages but that it leads to problems when it's time to scale up. Dropbox is writing their own Python implementation, and Facebook have their PHP to C++ compiler.

>how much would a company of Instagram's scale switching from Python to a lower level language like C++

I guess one thing you have to consider is

1. How many engineers do you have to hire to make Python "fast enough"

2. What does it cost to deploy a full C++ team? Is it easier for new programmers to learn & write safe C++ code vs. Python.

I wonder what is cheaper?

    s/C\+\+/golang/g

From the users perspective they're doing fine (haven't had any problems with Instagram performance).

Major con: Good luck finding people who want to write a web app in C++.

This is probably an extension of the "if developer time wasn't an issue"; at least that's a major factor for me (not being able to develop quickly).

It's important to remember that for a lot of network server purposes, your code only has to be fast enough to saturate your network interface, and after that there isn't much point in trying to speed things up.

Not sure how often that's the case in Instagram's use, but I've been using Python and Twisted to saturate network interfaces effectively since 2003 :-), and in those situations I've been more interested in the correctness advantages of a good statically-typed language rather than the performance.

time-to-saturation is also worth considering.

С++ is not exactly a comfortable language. While recent changes may be great, and there are quite nice libraries, the language still does little to prevent you from shooting yourself in the foot (and sometimes downright provokes it).

If I were about high speed, I'd take Java (which regularly rivals, and sometimes surpasses, C++ in speed), or Rust, or maybe Go. Other worthy contenders would be Haskell, OCaml, D, maybe even Elixir (unless they are CPU-heavy), though with these hiring becomes much harder than with Python or Java (and probably even C++).

Developer productivity is crucial to figuring out a business and then in the rare event that you actual get product/market fit, scaling is trivial (ie, no one ever failed to scale, not even Friendster).

Even when engineering time is not an issue, Python stays. Look at dropbox: they convert a part of they code to rust for performance reasons. Yet they have no plan to replace the whole Python code base, because Python is very convenient to use for no performance critical part of the app that are affected by it's dynamic nature. And they are not that numerous. In the end, it's not just time, it's also a matter of confort and even pleasure.

They should really show what they are doing with their CPU. How many requests per second and their characteristics? Generally Python (Ruby, etc) is terribly inefficient relative to the next tier of languages like Java and Go. At Twitter we were able to reduce number of servers by 10x and decrease latency by 10x by moving the web application from Ruby to Scala/Java with no architectural changes, essentially byte for byte (and bug for bug) compatible. My guess is they have done this math and would rather pay for more servers and take the latency hit rather than rewrite it.

Did you at the time compare it with a JIT implementation like PyPy (for Python, not sure what would be the Ruby equivalent)?

A JITing implementation of Ruby would by JRuby. Depending on JRuby's state at that time, Twitter could have used JRuby and gradually rewritten parts of it in Scala. Perhaps.

I think on reflection however we need to remember, with all our sprints and agile iterations, that just in Instagram billions of web requests are being served by just dozens of developers based on code written over 25 years now, following good practises and hard work - kudos to the Python Dev team on 25 years.

To be fair, that's more like "started 25 years ago". Very little of the original Python 1.0 code exists (especially given the great renaming when the "Py"/"PY"-prefix occurred.)

I like to think of the thousands of developers involved. ;)

For example, assuming Instagram uses gcc to build Python or any of the other components, gcc started in 1987.

Of course ...

If any Instagram people are reading this, please for the love of all that is sane, please I beg you add some kind of progress indicator for your API.

3 weeks in, I find myself loading my app's API client page to see if it's still stuck in a greyed out "go live" mode.

I've got an "Instagram API" search saved for twitter and StackOverflow to check checking if anyone else gets to slip through but others struggle just as much.

Just a little transparency and predictability is all that we ask for.

Wish they would share some of that profiling code...

I am digging into a gnarly bug in thousands of lines in python 2.6 right now and, well, a custom cProfile, C types support and auto-reporting sounds ... Lovely :-)

But I know that they will have their rough edge cases and developers banging their heads on the desk making this custom code for their needs.

But the grass does seem greener today :-)

Is Pypy used?

http://speed.pypy.org/

I wonder if they looked into using existing APM solutions before building out the profiler and monitor

Reply on news.ycombinator.com