Hacker News Reader

Comment by corobo | Hacker News Reader

I love these Twitter takes. They say something to get attention but if you look at it for more than a second it's.. it's just nothing data

Comparing a troubleshooting guide to running a site on a couple of servers is a bit too different for me. Compare it to a troubleshooting guide for those two servers, let's see how they stack up. No using any "ask {specific person}" either

Don't get me wrong, kubernetes is overkill for most side project level things. I don't disagree, I just like to see things knocked down a peg fairly!

Also as mentioned by this tweeter they use more than 2 servers anyway

https://twitter.com/shadowmanos/status/1434980544740306947

They could probably save on resources and maintenance effort if they switched to containers assuming this is still the same or more

> This is #1 in a very long series of posts on Stack Overflow’s architecture. Welcome.

View on HN

Here's the stats on Stackoverflow

https://stackexchange.com/performance

1.3 billion page views per month, 9 web servers, 4 sql servers

Stackoverflow is notable because they went down the C#/MVC/SQL Server route from the start, which meant much better performance per server. Thats why they make an interesting counterexample to the usual way...

> Stackoverflow is notable because they went down the C#/MVC/SQL Server route from the start, which meant much better performance per server

And also notable because everything is under an expensive license, so big performant servers is the cheaper option.

Edit: everything = Windows servers for their .NET app ( apparently in the process of migrating to .NET Core) and SQL Server

what do you mean by saying "everything"? just SQL Server?

Technically Windows licences, too, but maybe they migrated to Linux with their switch to .NET (Core).

Indeed, afaik they have been migrating long time ago

2018:

>This is the query pattern that caused StackOverflow.com to go offline yesterday:

https://github.com/dotnet/efcore/issues/13524

but on the other hand their tech stack site says: C# + ASP.NET MVC

instead of ASP.NET Core, but I don't think it is proof of anything.

They are definitely using ASP.NET Core, most likely version 3.1 LTS.

EF Core (like ASP.NET Core) ran on .NET Framework until version 2.1. Everything after that requires .NET (Core).

Updated my comment for clarity - yes, SQL Server and Windows for their .NET app. Maybe they've moved to Linux if they have finished their migration to .NET Core, but even if they did, it's a fairly recent thing.

> Maybe they've moved to Linux if they have finished their migration to .NET Core,

As far as I know they have been using .NET Core (or just EF Core? would be weird to go EF Core + .NET Framework, I guess?) on prod in 2018:

>This is the query pattern that caused StackOverflow.com to go offline yesterday:

https://github.com/dotnet/efcore/issues/13524

So maybe they used some things on Linux and some things on Windows - those that couldnt be ported to .NET Core back in the days

They're using .NET Core, and I think even .NET 5 in the meantime though I couldn't find a clear confirmation for that. But they're still on Windows, looking at Linux and containers for potentially in the future.

License costs are peanuts in traditional enterprise bugdgets, when mapped across the overall costs of developer salaries, contracting, designer agencies, travel expenses,....

Is StackOverflow a traditional enterprise for you? I don't think they have huge travel expenses...

Yes, it's a typical large software/SaaS company.

It's not about the travel specifically, it's that servers and licenses are the cheapest cost of any company when compared to salaries, especially for engineering and sales.

License costs are usually per-core, not per-server. It's the same whether you have a single large server or multiple small ones.

It used to be per-socket for Windows Server before 2016, and StackOverflow's infrastructure and architecture predate that.

You left off 10 servers: 2 Redis servers, 3 tag engine servers, 3 Elasticsearch servers, and 2 HAProxy servers. So, that's 23 servers in total, which is not a trivial amount, but also not a huge number, either.

What is really interesting to me is that they just have a very small number of each kind. It's not like they have 50 redis servers, it's just 2. 2 looks more like a smaller project ar first, but it's great proof that a lot can be achieved by doing engineering.

They are probably able to do it with little because they understand what they are doing.

There are too many developers and teams that don't understand the underlying principles and limitations of the technology they use. In a good case those developers get schooled by experienced engineers, but frequently they end up in an important position in some big enterprise software company and makes life miserable for many people.

I agree. This also ties back to the whole "a junior developer could clone Stack Overflow," or "I could do that in a weekend" mentality.

Sure, maybe a junior dev can make a simple forum, or you could make a web app that duplicates the functionality of SO at low loads in a weekend. But, when you're serving an average of 500 requests per second, you need to know what you're doing. And, IMO, this "knowing what you're doing" is the difference between knowing how to engineer a system and how to write an application.

> No using any "ask {specific person}" either

It's worth mentioning that the diagram is explicitly incomplete. The yellow endpoints are fixes but except for "END" the other endpoints are all either "unknown state" (i.e. "I have no idea what's broken") or problems that aren't addressed in further detail like "The issue could be with Kube Proxy" or even "Consult StackOverflow".

I'm not sure what a complete diagram would even look like but I don't think there's any way to infer complexity by looking at them in comparison.

People do tend to cherry pick, don't they? Most of Stack Overflow's workload is returning a blob of html for a given url, to a not-logged-in user. Where that html doesn't even have to be the most recently saved copy.

It's taking one source - look at how these people solved it! - and trying to apply it to others.

SO is relatively simple; it's basically customized forum software which is a solved problem that has been around for decades. A junior dev can build an alternative, and it can be built using tried and true solutions like MySQL + PHP, which are horizontally scalable with database sharding, read replicas, and maybe stuff like memcached to accumulate votes before updating the database or a CDN for caching static files.

Google has different problems and different workloads, and they have hundreds of times more applications with thousands of times more load. Apples and oranges.

> it can be built using tried and true solutions like MySQL + PHP, which are horizontally scalable with database sharding, read replicas, and maybe stuff like memcached to accumulate votes before updating the database or a CDN for caching static files.

> Google has different problems and different workloads

Which of these do you think most organisations most closely resemble?

I don’t think anybody would disagree if you said that you should use Kubernetes for organisations that resemble Google. But most organisations don’t look anything like Google. They look a lot more like Stack Overflow. So the “You don’t need this…” statement holds true for almost everyone.

Stack Overflow is also a) old system built before a lot of current management tech was available openly b) Very single-application centered.

EDIT: To expand on it - they had to manually build a lot of what could be much easier handled today, and thus had no incentive to change later. Interestingly enough, SO has a lot of moving parts distributed over multiple servers, even if all of those servers used to fit into 1-3 racks per DC.

The single vs multiple application centric thing is a point that's not made enough. Google is basically now a company all about spinning up new products, rather than just periodically adding a new piece of tech to an existing product. As a layperson, Kubernetes _appears_ to be about making it easier to spin up (and run) new products, so it feels more appropriate for Google-like org types.

There are also a lot many more applications one might need to run, especially when scaling beyond one-two people running all development on their laptops/desktops and not having a company yet - especially when one doesn't have valley-style funding.

Also these days you might not want to implement login functionality from scratch, or have better logging, monitoring, etc. and that might involve running more applications than just your LAMP stack.

>A junior dev can build an alternative

Of course junior dev can do it, the same way junior dev can make Youtube

it'll work as long as there's less than 100 concurrent users on SO and less than 50 4K 20min videos on youtube

Sometimes I question myself if people posting these "a junior dev can build this product" haven't really worked with systems at scale and all of the myriad of issues that scale brings or if they are being the usual hyper-optimistic-dev that isn't considering much past the proof-of-concept point of a product.

Either way is quite baffling how common this kind of comment is, almost a decade reading Hacker News and it pops up constantly.

It also doesn't matter if a junior dev could do it if they just don't do it

If it's so trivial, go be rich!

I remember this post from gregdoesit [1] as a good example of how much complexity is hidden under seemingly trivial interfaces. It's about Uber's supposedly bloated app, I believe that anyone who has worked at a scale where they had to support different national legal systems in their code would know that maintaining a large-scale product is nothing trivial.

[1] https://news.ycombinator.com/item?id=25376346

> a solved problem

Except 99% of forums run terrible software that doesn't perform, is not easily usable and won't work right on phones. That tells me it's not a solved problem at all.

Reply on news.ycombinator.com