Meta used monolithic architecture to ship Threads in only five months
infoq.com
infoq.com
So they, in fact, had already built out Instagram as a monolithic service based platform and essentially forked it, tweaked it, tested it, and shipped.
The title of the article is highly misleading, suggesting they shipped from 0 to full platform in 5 months explicitly due to the fact they were using a monolithic architecture as opposed to a microservice based one.
Still impressive, but the hard part of launching a Twitter competitor is not technical, especially when you already have a virtually infinite amount of already competent developers and infrastructure and when you don’t need the product to make money asap.
The hard part is launching it commercially and giving it enough momentum.
Like the adage says, I can code Twitter in a weekend but it will not be Twitter and it will not magically make money.
What's included in that 90%? Because I don't think the "What is happening?!" form and the timeline quite cut it.
Building a Twitter clone is an exercise that's frequently used in system design interviews. It's a popular exercise because it's a trivial system to implement to meet client-driven requirements, but the feature set becomes progressively more complex when scale, performance, and business requirements are thrown into the mix.
The top comment is funny in retrospect. "He already removed the login prompt which was a huge annoyance as a read only user of twitter." Twitter is now significantly more unusable as a read-only user, as you can't see beyond the first tweet in a thread.
His resignation four weeks later.
Text-book of how it takes more than just technical skills to get things done.
Public facing features are just the loss leading part of Twitter. It’s even probably a little portion of the company’s code base.
If it wouldn't be for the social part of connecting with long-lost folks from the past, I wouldn't use a service that behaved like a student project for decade and a half.
So actually shipping something working, and so fast, by same folks, is indeed impressive to me (not sure if same sort of basic bugs are there though, I am generally in direction of moving away from these 'social' platforms that actually in my view makes us as whole society more antisocial, we are simply built for physical interaction on many levels, we thrive with it and wither away without, even us introverts).
Arguably, they'd take another approach altogether in that case (keep some services in common and clone as few services as possible for instance), but it's still notable that they had the option to clone it wholesale in the first place.
> Despite the apparent advantages of reusing Instagram's platform for Threads (much faster delivery time), Malkani admitted the company introduced a substantial amount of technical debt that must be addressed in the future.
If they forked Instagram for Threads I bet the codebase is WAY more complicated than it needs to be.
There is a prevailing idea that large organizations do not have the communication capacity for many people to all work together, believing that more people require more meetings to coordinate, which at some point reaches a critical mass where any change requires more meetings than there is time in the day. Teams emulating the macro economy – a world where you can't get Google on the phone no matter how hard you try, where all you get is written documentation – is thought to be the solution to that problem.
So there would be something novel about multiple, disparate teams showing they can all work together under direct coordination while actually getting things done and not get bogged down in endless meetings. But, while the article is light on details, it appears that they actually did stick with a microservices model – creating a copy of Instagram so that the Threads team could provide an independent service without needing to communicate with the Instagram team...
I'm just confused on why these types of stories keep coming up recently. I thought the barrage of Sqlite was a lot, but "monolith vs. not monolith" seems to be even more frequent.
What exactly is being discussed in these cases? Most large systems don't boil down to pure "microservices" or "monolith". Look at the diagram on that page ... big clouds that talk to little clouds, big clouds that talk to yet another in-house KV store labelled "database" (that has boxes inside of it), big clouds that talk to cylinders, that in turn talk to more cylinders.
Then you get to the description ... it's Python running Distillery (custom Django), talking to "WWW" (PHP), with data stored in TAO using UDB. Add sharded MySQL, ZippyDB and Async (just to get some serverless in there?).
All of this with a term I've never had to use once in all these decades, "Server-Driven UI (SDUI)". After reading the explanation, this seems to be returning UI as JSON to be put together on the client, as to avoid waiting for a release cycle for UI changes? This sounds a lot like HTML and CSS.
What does any of this have to do with monoliths vs. microservices (or not monoliths)? I don't even know what I'd label this system. 8+ different technologies spanning server and client, but referred to as a "monolith"? That term has lost all meaning to me.
Funny they ran into timezone issues and a lot of technical debt. Now that I can relate to.
Yes, except for native mobile applications
This is not the feel-good "ye olde ways are always better" post that the headline suggests. This is a quick-and-dirty ship it by tomorrow™ and we will worry about building it properly later.
This strikes me as funny: what could "properly" mean to an industry if a successfully launched and operational piece of software serving clients globally somehow doesn't meet the requirements.
I think this talk of "proper" is just purism, its not grounded in reality or sound engineering practices.
The article should be lauded as kicking off a new era of practical engineering that rejects software purism.
That hardly meets all the requirements of a global and ambitious live service that has to move fast to gain traction against established, experienced and aggressive competitors.
It's a valid tradeoff and a successful launch, but it's ok to say it's "not properly built" for what comes next.
Again, what is this based on. Mostly on the fact that they reused existing parts to build a new service?
Why is this alone something to frown upon and conclude it is achieved by being "a mess under the hood"?
Maybe it is a mess, maybe it isn't. There's no reason to conclude this, other than the fact that there was reuse.
Hence purism.
EDIT: actually the software devs mention introducing tech-debt themselves, my bad.
You're basically arguing there is no such thing as tech debt, if a software works, then there can be nothing wrong with it. I would think this is obviously false.
They are not selling shrink-wrapped software. “Operational” is a moving, evolving target.
They could have definitely spent 12-24 months building the platform diligently from naught, but they'd have the same end product, just 12 months too late. The demand could have been satisfied by any one of several players in the industry (Tiktok, bluesky, LinkedIn, Twitter itself, Snapchat, a resurrected MySpace, etc), so rushing to gain first-mover advantage was likely the singular consideration for the project.
>Meta formed a small team to devise an approach to delivering a new service that would directly compete with Twitter in just a few months. Zahan Malkani, a software engineer at Meta, shared how his team was able to reuse the existing Instagram backend components, data stores, and large parts of the existing infrastructure stack but customize them to offer functionality comparable to what now is X.
And due to time constrains was even made sloppy:
>Malkani admitted the company introduced a substantial amount of technical debt that must be addressed in the future. The team is working on gradually separating the data models from Instagram's as the Threads service gains new functionality so that both platforms can be separated, but the process will take some time.
"Monolithic Architecture" // shows diagram with like 5 boxes on it...
Those five boxes represent a web proxy/server, an async system (likely analgous to an AMPQ server, Celery, etc.), a caching server, and finally the DB server. That looks like a pretty standard monolith at scale to me?
I wonder if most solutions could be implemented on top of Laravel/Django/Adonisjs without too much effort.
I’ll give you cloud, but my takeaways from HN in the last few years have been things like:
* Postgres is a reasonable choice for a job queue * SQLite is a valid choice for server db at quite large scale * k8s makes sense once you have 50+ devs * microservices address issues of team scale, not traffic scale
But what I, too, have seen is a greater emphasis on the boring, simple tools here on HN and I love that given all the experience and PTSD from the startup.
Minimum viable products only need to implement very basic functionality, just enough to get something out of the door. Limited tech stacks like Laravel/Django/Adonisjs can provide basic functionality, and sometimes a plain static HTML+CSS can serve that purpose as well.
Except that micro services are an organizational tool, which just so happens to have important technical traits. When you start to develop your MVP to actually do something relevant to your business, and you start to assign responsibilities and accountability to specific people running specific projects, you create a need to peel out functionalities and data sources and to isolate loosely couple them from the rest of the service. You start to have a need to have teams work independently and autonomously.
And that's where micro services come in.
There are plenty of system design exercises that ask you how would you implement Twitter, and the first iteration is a monolith with a RDBMS. It quickly grows beyond that. Why?
It's 4 orders of magnitude more than any customer of mine got in all the lifetime of any of their products. It's one order of magnitude more than what the mobile phone operator I was working for 20 years ago had after years from launch.
I guess that it means that an MVP is all we need. OK, they started with the Instagram code base but as you write, the core of the product is a pretty standard exercise. I did it a few times myself for some customers along the years, Rails and PHP.
Irrelevant. I'm talking about the feature set, which is not determined by the volume of clients being downloaded.
> I guess that it means that an MVP is all we need.
The clients need content. The business, in order to make money, needs way more than that.
For that you need data processing pipelines, user profiling, user segmentation, targetable ads, etc etc etc.
Not to mention infrastructure for moderation.
That's where the bulk of the complexity lies.
"Marketing" is where all the engineering goes.
You can make large enterprise backends or similarly large projects but I seriously wouldn't advise any of those three...
I wish AWS had an llm to just create all these stupid json files.
Simply put, with microservices and all the typical contemporary complexity you can't deliver a shit in 5 months.
Also - it can change
You’re picking a tool for the job at hand not declaring lifelong allegiance
I've got a few people I'd like to follow on Threads from my mastodon account, but I have the feeling this will never be generally possible.
It feels like something that will always be a low priority until finally a product owner decides that it can just be removed from the backlog.
https://engineering.fb.com/2024/03/21/networking-traffic/thr...
This goes to show that technology stack is not as important as devs (myself included) sometimes made it out to be. And that old, boring, battle-tested framework and technologies can actually deliver.
I saw several people drawn in their own tech stack drama, almost never creating a product.
I put some PHP together and have 34 companies paying.
It's a mess but I'd say that's more because it was my first real project, not because of php and jquery itself.
The good thing is you can spin up/down quickly, which is not always possible with some of those SHPs who want month-to-month commitments.
AWS especially will just suck you in with all the complexity. If you barely have an MVP you’re doing it wrong if you’ve ever thought about IAM lol.
Django is stable, well-known, and offers very long LTS. It's easy to find people who have experience on it, it is well documented and performs well enough for 95% of exiting websites.
You can quite easily convert DRF serializers to TS types/models, that you can then use in your preferred flavor of frontend framework.
Yes, and yet for many years now that's exactly the perpetual mistake that so many companies - both small ones and ones not in technology - would make because they assumed that if Google did it that way, it must automatically be the best.
Facebook PHP isn't the same as the standard PHP engine though - they've spent 20 years optimizing their own compiler and PHP engine/infrastructure:
> Overall, our experiments demonstrate that HipHop is about 5.5x faster than standard, interpreted PHP engines. As a result, HipHop has reduced the number of servers needed to run Facebook and other web sites by a factor between 4 and 6, thus drastically cutting operating costs.
https://research.facebook.com/publications/the-hiphop-compil...
Its no longer PHP heavy like it used to be.
But yeah, I used to get anxious reading HN and hearing about some new web tech stack. I thought my knowledge and expertise was falling behind.
99 times out of 100 the new stuff is nice, but not game changing, and the stuff I know is good enough.
Personally, I'm all for macroservices (no joke).
We use microservices, because we acknowledge that we don't know how final product would look like, and we mitigate risks associated with adding and removing unknown number of features.
Facebook was creating clone of Twitter, well known product, well known path. There is no need for microservices.
If you don't have a crystallized architecture yet, prototyping with a monolith is faster. You don't have to make the cuts (this belongs to this service) which are expensive to change later (unlike moving classes between modules).
Doing internal Java/.NET/whatever interfaces is easier than introducing external API (and associated infrastructure complexity, authorization, routing...), they are much easier to tweak. You have transactions, don't have to deal with a lot of asynchronicity, network overhead etc. For a prototype, I'd always rather do a monolith.
Probably not necessary to repeat here, but microservices add a lot of complexity due to asynchronicity, limited interfaces, and more complex error handling.
If you don't know how to design schemas (extensible, modular, ..) and don't know how to design coherent APIs around that, then yes, go Microservices to mitigate the risk. The risk here being the likely chance that you got the schema wrong and end up hacking around it in your "monolith" [almost always sic].
These have been pretty much figured out for about ten year now.
There aren't many interesting problems besides how to monetize it/make it profitable.
The only question is having the money to deploy at scale.
Wake me up when the next big thing arrives.
<?php include 'header.php'>
Hello <?=$name>, how are you today?
<?php include 'footer.php'>
Afaik, https://twitter.com/levelsio still works this way.