How meta built the infrastructure for Threads
engineering.fb.com
engineering.fb.com
MySQL, ScyllaDB, and RabbitMQ. A stack made in pragmatist heaven by devops angels. Now to find some developers who don’t hate it ;)
I have to say, I do love PostGIS. But for "this data absolutely cannot be lost or its game-over", I'd go MySQL every time. I can and have debugged MySQL replication on zero hours of sleep. It's entirely possible Postgres 16 is the better choice for new projects today, but I haven't had the time to try it out in production yet.
Man...times have seriously changed if that's the case. Good for MySQL to improve enough to get that recommendation.
* undetected hard disk in array fails
* battery in array controller fails
* disk fills up
* dubious backups, with no point-in-time recovery
* extremely poorly written SQL queries
* poory configured MySQL (in oh-so-many ways)
The top three (at least) would lastly cause replication lag, which would eventually trigger an alert. ... And yet we never lost a cluster. (And we far a lot of them!)
My team sweated blood improving processes and tooling, and then I spent a 6 month stint on database clusters (switching to GTID based replication and rewriting the ops config code so that they were all consistently configured and monitored).
Occasionally we'd get a new senior hire insist that PostgreSQL was a necessity, so we'd stand back and let them produce a proof of concept that stood up to the types of failures our MySQL clusters dealt with regularly, without waking oncall up at night. And it was always a bit of a joke by comparison.
The way Facebook apparently uses MySQL is not "MySQL + k/v store" but "MySQL is the k/v store". Forget SQL, they treat it as a key-value lookup and it's good enough for that. This also explains a lot of oddities coming out of FB engineering open source that at times look like they're trying to roll their own database on top of ... well, a database.
I wouldn't go to Facebook for architecture advice unless you have both the scale of Facebook and the technological baggage of Facebook. I'm not sure how transferrable this writeup about Threads is either given that Threads is so strongly tied to Instagram (although EU legislation seems to have motivated Meta at least to no longer require users to delete their Instagram account in order to delete their Threads profile).
That’s exactly what fb is doing. MySQL isn’t used as a k/v store per se but as the storage layer of the Tao graph database (which is also using memcache. So fb builds a database on top of 2 databases, technically). Source: https://engineering.fb.com/2013/06/25/core-infra/tao-the-pow...
source: I'm a former member of one of Facebook's MySQL teams, and among other things I worked heavily on the managed database-as-a-service tier. This was like an internal-only RDS, used by thousands of engineers for many different purposes, most of which involved SQL.
I'm a big fan of MySQL's scaling properties and some of the tooling (Vitess).
Do you see any emergent general purpose RDBMS achieving similar popularity of MySQL and/or PostgreSQL in the near future?
I heard good things about Clickhouse but it's a different tool (column based) from my understanding.
In all seriousness though, as a bootstrapped founder I'm not usually able to devote time to tinkering with emergent databases, sorry!
From what I've heard, Clickhouse is excellent, but it is OLAP focused rather than OLTP.
Keep it up!
Not nearly enough So please do that more often :)
Seriously I dont want HN to be one sided about RMDBS and I constantly submit a lot of MySQL related content and release notes. And there is still a whole world using MySQL for a lot of different reasons.
Especially now Oracle has finally updated MySQL release schedule.
Do they still use TAO?
Just wondering if they will ever open source TAO.
(source: https://blog.cloudflare.com/radar-2023-year-in-review-intern...)
It's in the top 5 overall apps across many countries in the EU. The Tech Threads and Photography community are especially good there.
I love all the quality of life features Threads is adding, but it severely lags FB/Instagram/Twitter in showing up to date information quickly.
What a nice way to say we don't respect our engineers. They should hide in the woods with their planning skills and not post about it
> The workloads commonly executed on Async are those that do not require blocking an active user’s experience with a product and can be performed anywhere from a few seconds to several hours after a user’s action.
But in reality, any managed queue or even a well structured DB table can be fine, depending on your scale.
I remember this being an issue a long time ago, where Lambda pretty much was a non starter for use cases where you're running 1000 independent tasks that each, on their own, is a small unit of work (just 1 or 2 database queries).
Long way of asking: is there a way to share a database connection between lambda functions or does each run need to re-establish / re-authenticate a new connection with the database?
https://www.jeremydaly.com/reuse-database-connections-aws-la...
It will get re-used across lambda invocations as long as the instance of it isn't killed which typically happens after a few minutes
This outlines creating the handler outside the function context to allow the client to be reused between invocations.
Apache Pulsar has been around for a while as a streaming system, and it does have a fairly significant Pulsar Functions layer which might make it a decent all in one candidate. (Pulsar in general feels like a much smarter lower-latency take on Kafka, allowing much more control/flexibility of where data gets backed/replicated/read from that makes it exceedingly more useful for "log/stream is the database" styles.) https://pulsar.apache.org/docs/3.1.x/functions-overview/
There's so many up and coming streaming things here. Apache Ballista. Netflix's Mantis. RisingWave. A million years ago there was SOA; a startup I was at used Apache Camel. Benthos. Others. Many of these will decidedly remain more streaming oriented, not encroach on FaaS space, which is fine, but I think we'll see overlap grow. Reductionist, but a function call is "run this code when a function name and parameters happen" versus maybe "run this code when an event type happens", which seems not so far apart to me.
More recently it seems they've added a login dialog to the site, but attempting to log in still just immediately prompts you to download the app[1].
Presumably once you've used the app to create and set up an account you can now use it to browse online, but that's not really "fully functional on web" and, relevant for the accusation of data-harvesting, still requires you to download the app.
Threads itself is launched in my region. Could you screenshot the front threads.net page you see (when logged out)?
> No app needed
Do you mean you're able to create an account without the app, or do you just mean that there's no app needed for browsing in your region (with no account, or after the account is set up through the app)?
The second screenshot is a form I can get to, but immediately after logging in it'll prompt to download the app to set up your profile.
[0]: https://timesofindia.indiatimes.com/gadgets-news/threads-mak...
I'm willing to gamble insane amounts on that's never ever going to happen.
It's been pretty chill over there, lacks the crazy that is Twitter
For me, it opens the potential for a federated social media protocol. Having a big player like Meta means everyone will likely benefit. Federated means we can move between servers without losing the network, allowing for competition in moderation, algorithms, and monetization
Threads has only been available for 6 months, they plan for full integration
If Zuck knows one thing, it's social media trends, it's what he's thought about his whole life
Once Threads became available in the EU I, like I assume many others, made an Instagram account from my FB account in order to check it out.
2 days later, and without having posted anything except perhaps a "Hello world", my Insta account was banned for generic reason (breaking the TOS, nothing concrete). I only found out 2 weeks later, as I hadn't even visited the site(s) again. Their "Appeal" button leads to a request to send a confirmation code to my phone, but no code ever arrives (Spanish phone, not a complicated country). There's no button or link to tell them that it doesn't.
I know of others who have experienced the same thing. Move fast and break things indeed.
They wanted to extract my phone number from me, and then also a photo/video after that to "verify" my identity, so I aborted the process, because I'm not going to feed the data slurping machine. I think that's the whole point of these fake bans, to extract personal data from you.
Twitter used to auto-ban me too in order to extract my phone number, and we know that Twitter illegitimately used phone numbers for advertising purposes.
To hell with the corporate-owned social networks. I want no part of them anymore. I'm sticking with Mastodon, which is free from this garbage.
It's unintentional on paper because nobody is stupid enough to put in writing that they intentionally make the ban mechanism very trigger-happy in order to collect more personal information.
On the other hand, it works in their favor, so there will not be any attempts at fixing this trigger-happiness either.
> The number of privacy reviews to add a data collection around bans
The number of privacy reviews to add a data collection around bans would be astronomical
Just like the privacy reviews that ultimately get them to continuously do things that are in breach of the GDPR and that even a layman with no specific legal/tech knowledge would consider creepy/unethical/illegal?
> I would feel bad for that engineer
That's the thing, there is no single engineer working on this. There is no ticket "increase the amount of personal information given by people" - as per my first point it would be a huge liability to put something like this in writing.
However, one engineer working on the ban system might have its KPIs tied to the amount of "bad" people blocked, where "bad" can be deemed as someone who got challenged and didn't subsequently complete or pass the additional verification (the thinking is that a legit user would have nothing to hide and complete the verification, and indeed many do). This effectively encourages ban trigger-happyness and indirectly increases data collection while being completely deniable.
Remember that these companies are not stupid/incompetent - there's lots of money at stake to get this "right" (in their own way - aka beneficial to their business model). They just have to pretend to be stupid/incompetent in order to get plausible deniability and deflect eventual litigation/regulation.
How big is this conspiracy? Hundreds to thousands of engineers leave Meta every year, and 0 have blown the whistle on this. The actual FB whistleblower from 2023 didn't allege anything similar to what you're suggesting.
As I also value freedom of association, I am perfectly alright with them refusing to "do business" with me because I do not find the denomination and terms of the transaction acceptable.
I do not really have any inclination to want to compel them to serve me by force of gun, either.
I think both they and myself are better off in this scenario where we just choose to not do business with each other and I find some other business that has more acceptable terms - or I just go without, these services aren't as essential as some people think.
They just were unable to send the code to it, so I could not proceed with whatever appeal process they have in place. That said, if the next step was asking for my face, I'd have probably stopped right there like a poster said. They do have photos of me (Facebook) but it would just be too much.
I can only dream of being able to cite an article like this on my impacc...
>Threads was developed in an environment more akin to a startup.
OK sure, a small group of people broke the model of development and did something new from the ground up?
>Threads scaled successfully to 100 million users without any major downtime thanks to Meta’s underlying infrastructure and engineering foundations, which were critical to the successful launch.
Ah, no. All of the underlying hard parts were done already for them. That's not a startup.
[1] https://engineering.fb.com/2023/09/07/culture/threads-inside...
0. Exaggerating some of the bureaucracy I've seen in larger orgs
Personally, it has become my preferred social media platform. It's overflowing with content and I check in on it several times a day, and have been really enjoying the community that is emerging.
It DOES however have a big porn bot problem, but I don't think that is unique to Threads. Hopefully, they can come up with a way to minimize it soon.
when you push it relentlessly in the facebook app its kind of hard for it not to become #2 overnight.
I give threads eight months before its either riddled with the same trash boomer content as Facebook, or devolves into X in a desperate attempt to emulate its former success. This is less of an innovation for Facebook and more of a convenient pivot now that theyve run out of gas entirely as a social media platform.
LinkedIn is beating them all, significantly. Instagram is next, which surprised me all little. After that it was X and then finally YouTube, which didn't have any content until September so it will likely/hopefully overtake some soon.
Threads AND Facebook have been kinda terrible. We just shuttered the Threads account and probably getting ready to do the same for the Facebook account. Threads just picked up some followers from Instagram after the initial launch.
Mastodon has a low follower count, but solid engagement among the followers who are there at least. Keeping that one around longer.
We may try Bluesky at some point and I'll continue remaining open to trying new things as they come out, but nothing made me optimistic about Threads from our trial so far.
Here's the report: https://twitter.com/CodesCarolina/status/1744410822720303348
Seems like your metric is "follows for the sake of follows", which is a vanity metric that's easily gamed. What conversion actions are you measuring, and how are they doing?
Engagement is best on our email list, which is nearly parallel with LinkedIn and we’ve got a 40% open rate there.
These numbers are essentially from setting up our Substack and broadcasting the posts out to various different platforms.
As somebody with no idea what they are doing and limited time, what should I be measuring?
On X I get mostly informative ML and robotics content. Of course it took some curation and liberal use of the "Mute Words" feature to get there. But I just don't think that content is there on Threads.
My go-to example of why I don't like what's happening on Threads is Jeff Bezos's last post: https://www.threads.net/@jeffbezos/post/CyFInFjrRxs/?igshid=... The replies are an absolute cesspool. Everything people used to accuse Twitter of being and more.
This was also my experience.
Why do people enjoy this stuff.
FWIW, you don't have to use Threads' "For You" feed (the equivalent of X's algorithmic feed) at all. But if you like the kismet of seeing other posts, blocking liberally will sort that out.
It feels like the whole purpose of Threads is just one-upmanship against Elon Musk, and as such, it's just a hollow shell of a community.
Pointless rage-bait Threads content regularly comes up on Instagram and I find it a turn-off for the new social network.
Only NPCs expect good, uplifting content to be handed to them without work.
This applies to both X and Threads.
If there weren't pressure to use it for my small side business I'd have given up on it almost immediately.
And to be clear, I deleted my Twitter account shortly after Musk's purchase. I'm not an X "shill", though to offer the counterpoint when I see someone "shilling" Threads I assume they have some iota of success there and really, really want it to be a thing so they'll constantly tell you all about how it ranks in downloads.
Thread's algorithms are absolutely fetid garbage. Despite following a couple hundred good accounts and trying to like content I enjoyed, my for you was endlessly filled with almost an inverse set of what I want to see. I didn't engage these posts but instead would just mute/block copiously. It would redouble the same. "Tech" Threads recently has been 100% occupied with Meta employees bitching and whining about Apple, which grows super old. I could switch to Following, but so few people actually post there that it's just sadness, and the few who do post frequently are so obviously desperate to try to make a social media "presence" that it's just...the opposite of not cringey. And again it is shocking how much content on Threads is about Threads, or about Twitter. It's just incredibly boring.
And this is the honeymoon period! If this is the best it can offer, what happens when Meta starts actually turning the screws?
The major problem is that Threads will surface and even recommend (via your Facebook feed as well) threads that people you follow have responded to. Great in theory, but on Christmas Day I opened up FB to find it recommended this highly toxic anti-gay post in the middle of all my friends' posts about the holiday. It was because someone I follow on threads took the bait and commented on this post.
I've had far too many instances of this sort of behavior in my life and the only options available were to unfollow everyone, constantly block these abusive actors, or leave. For me, it was absolutely not worth staying, despite the up sides.
I'd encourage people to give Threads and Bluesky a try. Both don't have a CEO that amplifies conspiracy theories and falsehoods.
That’s like saying it’s impressive I made a million dollar company after my dad gave me 2 billion.
Just was pointing out user numbers and that the average Hacker News commenter’s bubble is not representative of how most of the world uses social media.
And now they have reached 100M MAU which shows that people are coming and staying.
This statement is meaningless without knowing what the product, revenue and its Total Addressable Market are. 100M MAU out of 1.5B TAM is not attention-worthy. Twitter at it's peak had 550M MAU, so Threads doing close to 20% of that in less than a year is impressive.
The current MAU is calculated by counting the users who see threads on instagram and accidentally click on it - which is A LOT.
I would still consider Threads dead because you don't hear about it. You often see screenshots of tweets or links to tweets on here and other sites, this tells me that people are actively using Twitter/X and engaging with it. But as for threads? I haven't seen a single screenshot or link to Threads outside of ads you get on Instagram about it.
What I see is that Threads is on life support and the life support machine is Instagram, if not got IG, Threads would be 100% DOA.
(and also a shameless plug: I featured this article in my last newsletter https://bigtechdigest.substack.com/p/most-read-tech-articles...)
I'm sure harvesting data for LLMs is a great side benefit, but if that was their main goal they would probably just run a crawler. They could even crawl ActivityPub without having their own instance if they wanted. It is a public API. They would just miss out on private posts which is probably a small fraction.