Hacker News Reader

Comment by endisneigh | Hacker News Reader

It’s fascinating to see companies solve the same problems over and over again.

I hope in the near future these types of problems are “definitively solved” and more effort can be spent engineering novel features.

There’s no way to do this, but I’d be interested to see if there’s a company that just has an extremely simple stack that is very performant and popular.

I’m talking: their entire stack is Citus sharded over a few locations running two beefy servers per continent they have users in, that’s it. No dedicated search, logging or anything. Just dump everything in Postgres. If logs are lost then so be it.

A lot of the problems at scale are problems of our own creation. For example is strong consistency really necessary for tweets? Do you really need to see a followers latest tweet instantly?

Uncheck some boxes and the engineering becomes a lot simpler, very fast.

View on HN

> For example is strong consistency really necessary for tweets? Do you really need to see a followers latest tweet instantly?

The problem is that "strong consistency" basically means your system behaves in predictable ways that you can easily reason about. If you give it up, you can see all kinds of failure modes that go beyond just seeing stale data.

Suppose a user first sets their profile to "private", meaning only their trusted friends can see their posts. And then, having seen that that operation succeeded, they make a post that contains personal, sensitive information. You had better be damn sure that no user-facing components of your system can ever observe those two events in the wrong order.

I don’t disagree at all with what you’re saying, but it’s all about expectation management.

People should expect that when they mark their account as protected it is indeed protected.

Things need to do what they say on the box

If you threw strong consistency out the window and kept low latency you're still getting the "make profile private" event well before the user has navigated to the post page and composed a tweet.

The problem is ensuring that event has been propagated all around the world so you aren’t displaying now-private tweets. That’s why we need strong consistency.

I have spent much of my time at this company on this exact problem.

"and kept low latency" is not a reasonable assumption in the face of real world networks.

If the latency is bounded, then you have strong consistency. Just add a sleep(5) call after every write to make it linearizable.

What if the process interrupts before it could sleep(5) and restarts...

Still linearizable under the insane assumption that the db is always “low latency” which implies it never goes down. The write either happened or it didn’t.

Latency is never bounded, in fact that is what the P is in CAP.

Yep. It was not a serious suggestion. Without partitions, everything would be easy so it’s kinda silly to assume low latency.

Yes, what I’m saying is that it wouldn’t say that until it’s true. There’s no disagreement here.

One problem is that at a certain point you can’t just lose logs - for example, somebody might pay you per ad impression or click. Losing logs can mean losing evidence that those impressions/clicks happened. Logs aren’t just a debugging tool, they can be a critical legal CYA.

I’m not saying that people don’t overcomplicate projects (they do) but a lot of simplifying assumptions can break down at scale.

But why would one use log as source of truth ? If the info logged is important for the business, probably it should be stored in a DB (say).

Logs are stored in a database, just not usually a relational one.

True, logs do needs to be stored somewhere. What l meant was: if the info is so important for the business, that detail should be stored in a system that provides a longer durability than just logging it.

Though I agree, I think the parents point is that you don’t necessarily know if it’s important so you have to have some general record keeping with respect to “activity”

> just logging it.

This is the misunderstanding here. With a sufficiently complex system, they are never 'just logs'.

Hmm, I think that anything you do not consider ‘just logs’ should be somewhere in your main database instead.

That's backwards. DBs you can lose because they can be reconstructed from the logs.

Seems we are referring to two different things. What you mention references DB journal logs, and my comment was about application logs (like debug, info, error message etc.)

You can even handle impressions/clicks/conversions in a lossy system, you just have to bound how much loss you're willing to accept. The counterbalancing factor is how expensive you want your logging infrastructure to be -- the more consistency, the more it costs. The other trade off is availability: what do you do when the logging backend is temporarily down? Return a 503 to the user? Or accept that you're not making money on that impression and move on?

Storing everything is fairly simple and cheap. The complex bit is retrieving/aggregating/searching.

As you enter larger corporations, you see more Resume Driven Development. Overengineering isn't something to be avoided. It is the purpose of your 3 to 5 year stint there. Management rewards it so everyone participates in the circus together until one day the board fires 80% of engineering and hires contractors.

How to prevent this though? From a game theoretic perspective?

I'd suggest you look at Stack Overflow engineering posts. Their architecture is about as simple as you can get for a service their size. A bunch of web servers running plain old ASP.NET MVC code (recently upgraded to use ASP.NET Core). A handful of SQL server machines for the database, simple sharding strategies. Some Redis machines for caching.

Twitter’s real time aspect is a core value proposition compared to other social media platforms. Nothing tells you what is happening right this second as well as Twitter. It is faster than earthquake shock waves. And people rely on that speed for multi hundreds of millions of dollars in revenue.

This feels heavily overstated. Especially when you consider that the vast majority of "influencers" and such actually schedule their tweets.

I agree that current tweets area important, of course. But I also acknowledge that it is just as often that i will be scrolling through the "In case you missed it" section of tweets. And, I have no idea how rapidly those actually made it to my timeline.

Is it? Especially when there is an xkcd about this exact phenomenon? https://xkcd.com/723/

Is it overstated? Since my timeline constantly shifts under my scrolling, often showing me it of date crap, yes. Worse, many posts are ads or trending items that.. no, you would not know if there were seconds of lag in them getting processed.

You could almost argue they the small like counts have to be strong consistency. But, I often get notification of a like well before it shows on my timeline. So it is clearly not strongly consistent.

You have to take into account that the people that Twitter makes the most money off of for real time functionality are not integrated through the web interface.

I'm no fan of Twitter, but I use it constantly to see the latest on happenings around the world.

But would you know if there were seconds of consistency catching up? How?

How would an influencer schedule breaking news and live on the ground footage of whatever thing that is happening in the world?

Additionally, Twitter is typically a poor place for influencers. Keep em on snaptokgram

Oh, and the companies paying hundreds of millions of dollars are not integrated through the web UI

The previous POTUS made important policy announcements on Twitter first. It would be very difficult to understand what was going on without fast, consistent tweets.

With slow, unreliable tweets he might have had to be filtered through normal channels, which would have meant they came out at more predictable times like “press conferences” after being discussed by the people involved in that whole process, instead of “I watched Fox and Friends and now I’m pissed off about whatever they wanted me to be angry about today”. It would have been much easier to understand things, that shit was constant chaos.

Please give an example of a circumstance in which this was true.

Trump announced a ban on transgender soldiers via tweet like 1 day after giving a heads up to the Defense Secretary who was on vacation at the time. Nobody else in the military was prepared for the policy change and they were left scrambling.

Multiple times he fired/hired senior members of his administration via twitter, including defense secretary, secretary of state, national security advisor, chief of staff, ...

> Uncheck some boxes and the engineering becomes a lot simpler

I think you're oversimplifying to the point of nonsense. In my domain reliable logging is an absolute must due to regulations and the fact that we deal with people's money. Pretty sure we're not the only industry that can't afford approaching logging as you painted it.

Sure, but most industries aren’t regulated to the extent that you would need logs.

Reply on news.ycombinator.com