Hacker News Reader

Comment by colesantiago | Hacker News Reader

> Instagram uses Python (Django) for its backend. By using the same backend for Threads, we could leverage a lot of the existing tech stack for Threads and reuse most of our existing data models, business logic, security features, and server infrastructure. This also meant users could sign in to the app with their existing Instagram account, making it super simple to onboard and set up your Threads app.

Looks like Python and Django is extremely scalable at Facebook scale, even after using Threads I didn't realise any downtime at all.

It would be possible for Mastodon to learn from this so that they too can scale to 200 million users.

View on HN

The only lesson to learn from this is that given exceptional resources and a long list of talent any stack can be made to scale.

That doesn't help most projects in the world.

> The only lesson to learn from this is that given exceptional resources and a long list of talent any stack can be made to scale.

Exactly. IIRC, Facebook itself is/was a giant ball of PHP, and they had to do all kinds of crazy things to make it scale, up to and including writing their own compilers and VMs.

That doesn't say anything about the suitability of the scalability of stock PHP.

I think the lesson is, avoid rewrites and don’t break apart your monorepo; have teams improve the platform while your product engineers go full-steam ahead. Compare with Twitter which spent 2 years in the woods rewriting to Java while FB kept growing, giant ball of PHP and all.

You’re getting downvoted for this, but you’re not wrong.

Just keep moving forward unless the sky is falling is 100% the lesson to be learned here.

In the worst case, you'll fail to scale due to technical shortcomings like Friendster and become forgotten while later entrants like Facebook eat your lunch.

There was a time when "being web scale" was a real concern. That's one reason companies like Google make such a big deal about it in their technical interviews. Now most of the hard work has been done and exposed as services so today even an untrained teenager can stand up a web-scale service.

Python apps using popular frameworks (not all frameworks) are generally (not always) easy to scale. The app itself is mostly just "spawn another instance somewhere" - storage and sessions and all that can be complex, but that's not really impacted by the app framework/language.

(I use django in production myself, nothing against it)

It's probably worth noting Meta made their own JIT'd version of CPython just for Instagram/Threads. Maybe mastodon devs can use their interpreter, but Meta's human/compute resources are magnitudes more than mastodon, the lessons from Meta might not be very applicable.

https://github.com/facebookincubator/cinder

Isn’t Mastodon written in Ruby?

Yes, I wasn't sure if parent was suggesting they switch to django, or simply "using what facebook learned" - I didn't want to imply that there's no way Mastodon devs couldn't use Cinder to their advantage.

I do however highly doubt Cinder would be of use to Mastodon at the current moment.

recently they made cinder public… this is a JIT’d version of cpython

I don't use Python or Django anymore but I'm always happy to hear about it and kinda surprised to learn it's still being chosen today for massive-scale social media sites. To me, Django was my go-to for cranking out a usable MVP in the shortest time possible. It made it so easy, I could define a DB schema and it would generate these forms with validation built in. I kinda miss those days.

What you're saying is correct, but oh boy have I worked on Django apps that have had awful security issues because of this. "look, all my endpoints are there, user A must only be able to reset their own password... right? ;)"

Also you don't have to miss those days! You can still use Django or a similar library? I still use Flask, it is great.

I'm pretty deep in the Typescript world these days but hey, maybe one day I'll make my way back into Python!

I sort of wish that Go, Rust, Zig has these kind of frameworks like Django, Laravel and Rails.

I believe there is an opportunity here to make the next Django in these languages.

Each of those absolutely do have them. Here is a list for just Golang: https://github.com/avelino/awesome-go#web-frameworks

There are a lot of things that goes on in production to scale any service to handle that level of requests. But Django as a web framework is good at what it does. It comes in with most things that one needs to put web apps in production.

We recently open-sourced an LLM apps platform (https://github.com/trypromptly/LLMStack) that is entirely built with django as backend (drf for APIs, channels for websockets and reactjs for frontend).

Mastodon has such a different system architecture—at a business level—that I think many of the scaling challenges are entirely different. While my experience with Mastodon is as a (very satisfied) user, in general scaling challenges are as much figuring out how to scale the interactions between systems as they are scaling the individual systems themselves. That gets incredibly more difficult when those individual systems are spread out among various organizations with different priorities, philosophies, technical abilities, and resources.

This is largely due to Meta maintaining an internal performance-oriented production version of CPython: https://github.com/facebookincubator/cinder .

Python/Django (and most other frameworks) isn't the problem with scaling. It's following the 12-factor app rules.

Python is known to be pretty wasteful in terms of compute performance, but is the compute grunt of a given node a big deal in these kinds of networks? I assumed it is a big communication problem. Python shouldn’t hurt there, right? It isn’t like a Python-generated packet goes through the network more slowly.

> t would be possible for Mastodon to learn from this so that they too can scale to 200 million users.

I mean possibly, if it was designed to be a centralised bohemoth with a staff of 600 to look after it.

Its decentralised by design, much like the web used to be.

So did they setup a separate backend for threads or did they just extend the current instagram backend to support text posts?

These systems should not scale. Giving everyone free broadcast capability is a sign of mentally bankruptcy.

Its like giving every neuron in your head broadcast capability. What happens when they all fire at the same time? Or like giving everyone in the world a radio transmitter that can broadcast on any frequency. What happens when they all transmit on the same frequency at the same time?

These idiots are scaling things because its become possible technically. There is no other logic to whats going on. And its producing/enabling massive amounts of useless noise. The UN report on the Attention Economy says 0.5% of content on the net is actually consumed by anyone.

> These systems should not scale. Giving everyone free broadcast capability is a sign of mentally bankruptcy. ... These idiots are scaling things because its become possible technically. There is no other logic to whats going on.

I agree, but I don't think it's a sign of "mentally bankruptcy," but an indication of the flaws and limitations of contemporary ideology. They're not just doing it because it's technically possible, they wouldn't even try and no one would sign up if our ideology didn't say it was a good thing to do.

Reply on news.ycombinator.com