How Facebook Keeps Messenger from Crashing on New Year's Eve
spectrum.ieee.org
spectrum.ieee.org
One thing I am curious about: how does it compare to how the old postal systems used to handle Christmas and new year loads?
On a lighter note: perhaps you can predictively generate and cache messages at the receiver's end based on their contacts and their style of communication. When a sender actually sends a message, just send one bit across, and the local cache gets flushed and displayed :)
I'm reasonably sure that UUCP + things like INN were "batching messages per destination" for efficiency a long time ago. Nowhere near the same scale, obviously, but the same kind of concept, no?
The raw numbers (users, bandwidth, latency) would be quite different.
Even by reducing a message content to close to 0 bits, I doubt you would get that much gains. Most common messages would probably be things like "lol" or an emoji, which in utf-8 is 24 to 32 bits.
But There'll always be metadata associated with each message: message id, sender, receiver, timestamps, etc. which are unremovable. If those first 3 are typical GUIDs, this is already 3*128 bits. This will put a minimum on the entire message size , and more importantly the processing time to simply route the messages is what is costly in terms of CPU time. Then there's the different receipts mentioned in the article, adding a lot more traffic.
Facebook user ID's are 64bit and I doubt that the message ID needs more than 64bit either. Timestamp is typically 8 bytes, so 64 bits as well. So, you're looking at 256 bits of message header.
Offtopic- I hate those features to begin with, but I know I'm the product not the customer, and those features are to keep people on the app.
To me, "seen" is a feedback mechanism. When I send a message to you, I don't always need a response, but I do usually want to know that you saw the message -- sort of an ACK.
But, but, but, it says 'instant messaging' right in the name.
You should reply the _instant_ you get MY message!
Unfortunately some people have the same attitude even towards email...
Maybe I should type a response the moment I receive the text across my phone alerts, but I want the freedom to defer it without having you think I'm ignoring you (even tho I am)
[1] https://easylist-downloads.adblockplus.org/message_seen_remo...
I think the "seen" feature is a net positive though. Almost everyone likes it from the PoV of the sender.
Lets just say it was not my favorite change, and not just for the privacy reasons under discussion.
Isn't that a preference in the notifications setting? "Show Previews" gives the options of "Always", "When Unlocked (Default)", "Never". Perhaps it's just defaulted to "When Unlocked" after an update?
If someone is obsessed about another person, they're going to get the same kind of dopamine rush simply from seeing that the other person is online and active that most of us get from receiving new messages in our inbox or messaging apps. It's tricky to explain, but as you draw closer (or feel like you draw closer) to someone, the urge to be connected and know that they're present becomes overpowering. Moreso if that person is in any way encouraging this kind of dependence / attachment.
It's particularly bad when apps allow this kind of 'presence monitoring' without trying to deal with the obsessive behavior that it causes. Facebook, for example, clearly has the data and the ability to inform both parties of what is happening. Ironically in my case it suggested one of these people as a 'Close Friend'. It seems so much worse if the service provider is (at any level) encouraging these malign use cases - for the purposes of 'engagement', for example.
Fortunately I believe I now have the distance and levelheadedness to see how this all works (in terms of my own psychology, the relationships involved, and also the service providers who intermediated the communications), but it's something I wouldn't wish anyone else to suffer, and we could be a lot more careful about the way we design apps and services to avoid these kind of psychologically damaging situations.
The talk entitled 'What is Good Technology' from the 35c3 congress makes some very good points along these lines, in terms of what we can do improve our technology product decision-making process to avoid causing pain and damage to cultures or people we might not otherwise understand: http://streaming.media.ccc.de/35c3/relive/9965
read two days ago
Yikes.
I had a 3rd party client (aMSN) that would popup a chat window as soon as someone would start typing an initial message to me. I would say "Hi" to them first and they would thing it was a big coincidence that they were trying to talk to me at the same time.
They kept incrementing the server protocol version without updating the client. Eventually all the clients just stopped being able to connect. Strangely, for a long time, only the version bundled with XP stayed working, until eventually it too could no longer connect.
FYI, I don't think that is possible in Slack. You can choose to hide typing indicators in your view, but AFAICT you can't prevent other people from seeing when YOU are typing: https://twitter.com/SlackHQ/status/1057288342671368194
I guess someone like me could make a browser add-on, too. Though I don't know how many people use the browser client vs. the app.
If you run a proxy you could also block those requests.
Likely against their ToS (https://old.reddit.com/r/programming/comments/9bc6gi/bye_bye...)
> Hi
(waiting) (waiting) (waiting)
< OK Hi
(typing) (pause) (typing) (pause) (typing) (pause) (typing) (pause)
FFS hurry up already
They perceive it as less confusing.
/edit: been informed by some of the people in the picture that technically there are 2 teams there: Messenger Infra and Messenger Foundation
I wouldn’t be surprised if on NYE services like chat take priority and get extra resources from background jobs that can wait a few hours.
My guess is they're doing both.
Anecdote time. I worked at a company where one project was over-provisioned on dedicated hardware and another auto-scaled in the cloud. The over-provisioned project was much cheaper, had significantly better response times and was easier to manage. It was load tested to handle over an order of magnitude more traffic than the all-time-peak and even though fully over-provisioned, it was cheaper than the baseline usage (and slower, and harder to manage) cloud solution.
There was a very long period of time where SSDs were commonly available from everyone but cloud vendors. For some workloads (like databases), that resulted in a massive difference.
This is still true for bandwidth. You can look it up yourself. Off the top of my head, in the US, you can find a server with 100mbps dedicated port for < $300. But that AWS bandwidth is over 3K. So that's less than 1/10th AND you get a relatively [relative to what you can get on AWS] powerful server vs just the bandwidth.
Back when the C4 instances were announced, I ran unix bench on them as well as a dedicated i7-4770. You can see the i7 was quite a bit faster, and, if I recall, was less than half the price
https://gist.github.com/karlseguin/5a6a45ace2048545b6c3 vs https://gist.github.com/karlseguin/a659ef87b3a4a5d590e9
I think database workload is still where the average app would see the biggest difference. A properly configured server with a battery backed raid adapter and proper dual network NICs will blow RDS out of the water for raw latency and cost and, most noticeable to me, consistent performance.
Unfortunately, fewer and fewer companies seem to be offering servers with BBU and dual NICs. And those that do are charging more...so that's also helped close the gap. IBM really screwed up. They wanted to compete with AWS so tried to turn Softlayer into an AWS clone, rather than focus on what Softlayer did better and fix the issues (automation and ddos mitigation come to mind).
However, I think your rationale begs a question: how much of "the automation and services that major cloud providers have" (scare quotes only for readability) is needed by runners of metal? For instance, autoscaling can be mooted by overprovisioning, which would still incur only a minor cost increase. Multi-region is similarly cheap. A lot of the remainder seem like productized functionality that is fairly implementable locally if desired.
We did two games, one over-provisioned on metal, the other on auto-scaled cloud based infra.
The cost is significantly higher. We ended up with a hybrid to control for cost. But our needs are long-lived sessions which does not fit the elasticity of the cloud model well.
https://www.facebook.com/notes/facebook-engineering/chat-sta...
Messenger is (also) a finance system. You can send money via Messenger. You can purchase products directly in Messenger. It's had all that for more than two years now.
are you on a web browser? are any of your extensions mucking things up?
1 billion 100 Byte messages sums up to an almost trivial 100GB. This might be a technical challenge for the neighborhood's web admin but not for any real company.
P.S. There's no such thing as a 1985 GSM network. That phrase just makes your comment look even sillier.