Hacker News Reader

Comment by yongjik | Hacker News Reader

Before anyone wants to criticize Stack Overflow, consider their server footprints as of 2016 [1]:

  4 Microsoft SQL Servers (new hardware for 2 of them)
  11 IIS Web Servers (new hardware)
  2 Redis Servers (new hardware)
  3 Tag Engine servers (new hardware for 2 of the 3)
  3 Elasticsearch servers (same)
  4 HAProxy Load Balancers (added 2 to support CloudFlare)
  2 Networks (each a Nexus 5596 Core + 2232TM Fabric Extenders, upgraded to 10Gbps everywhere)
  2 Fortinet 800C Firewalls (replaced Cisco 5525-X ASAs)
  2 Cisco ASR-1001 Routers (replaced Cisco 3945 Routers)
  2 Cisco ASR-1001-x Routers (new!)

Assuming they aren't lying, this is the entire list of machines needed for a site used by the world's developers. That's no small feat. Many companies whose developers blog about their "best practices" would struggle to achieve 1/10th as much efficiency.

[1] https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar...

View on HN

I hope this is obvious, but best practices can often intentionally sacrifice performance for the sake of other aspects. The fact that breaking them can give performance gains shouldn't come as a surprise and can be the right thing to do.

My reasoning is that they should only be broken if you understand the reason for them, how it applies to your situation and the implications (or if nothing's at stake, which can be a great learning experience as to why those recommendations are made).

The part that's not obvious to those applying best practices is doing so without such a similar understanding. It is a shallow appeal to authority of the 'best'.

performance has almost always been about violating best practices because a simple happy path isn't usually suitable for hot traffic (e.g. bulk insert vs insert into).

Back in the early 2000s when the traffic just started to hit people were de-normalising databases just to make sure stuff could even work.

Even before breaking them, I think one of the bigger problems with best practices in practice is, that many people have forgotten or never tried to understand why they exist in the first place, what problems they are trying to solve and how, their scope and what the trade-offs are you are making by implementing them.

Instead they get treated as magic wands swung around promising to solve all the worlds problems if you just believe in them strongly enough.

On the other hand, I worked for a company that seemed to follow Stack Overflow's article (sparse tests, unmockable static fields) that made everything a nightmare and slowed velocity almost to a halt, and provided a brittle customer experience.

I can quite believe that caching SQL results saved time (hardly not a "best practice"). The waters become murkier when you say you can't afford basic abstractions to make things testable.

They provide no statistics on how much time their "optimisations" actually saved. Likely, their optimisations wouldn't have made a difference, as they would have been JIT'd away anyway.

I am very much opinion that if you're writing code without some sort of test, you're writing legacy code.

While they do not provide any benchmarks in the summarizing article, I can guarantee that they have done some (read: alot) profiling. I follow both Nick Craver and Marc Gravell on Twitter, and they regularly post benchmarks for optimizations done at SO, in the .net world they are some of the most knowledgable people on performance optimizations.

> (sparse tests, unmockable static fields) that made everything a nightmare

I had the same thought. I've noticed over the past few decades that most developers migrate toward static fields (i.e. making everything global) not for performance reasons, but just because it's requires less up-front thought. And when the only thing that shows up on your year-end performance review is "did you hit the date?" this is almost understandable. Maybe the folks at SO really are that good - they can code everything right the first time and there's no particular need for unit tests, but I'm used to finding bugs that have been in the code for YEARS that would have been trivially caught if the code had just been run even one time under observation. I'm also used to code that's so monolithic (statics that import other statics that import other statics) that it takes months for new hires to get any sort of proficiency in - and that's if they're really experienced to begin with.

https://stackexchange.com/performance

There are some more details on this page, including the typical CPU usage on those servers.

What I don’t understand is how they can stand dealing with IIS. IME it’s so extremely frustrating to use, especially compared to Nginx.

It wasn't really an option when they started. Nginx couldn't and still can't run ASP.NET application. You could maybe put the application in a container, or run it as directly with the new versions of .Net, but that requires reworking the entire application. There's not really a lot to gain, not if you already know IIS.

Ah. That makes sense.

I wonder to what degree they have automated the configuration of IIS…? From my POV it seems really tricky to automate. But I am by no means an IIS expert, I just have some clients that use it.

You can automate IIS configuration with Ansible, that's what we do.

Mostly the issue is documentation, you sort of have to know what things are called, but you can get around that by using win_iis_webapppool and get the configuration from a running IIS server and then work out which properties you can set.

There might be things you can not automate, we don't use that many advanced features.

Edit: I wasn't being fair. Ansible, is actually pretty good at managing Windows servers. It's just different from managing Linux servers.

I'm guessing you just don't have much experience with it? As someone who's been using IIS for 20+ years, it's all very straightforward to me.

Anything is "straightforward" after doing it for 20+ years. Just ask a COBOL programmer.

It's no small feat, but the nature of SO is very cache-friendly (content focused, reads >> writes). If you're building a modern SaaS, dealing with the same amount of traffic as SO, you might find a hard time with the same setup.

Reads are much more frequent than writes in the vast, vast majority of programs.

Am a firm believer that a billion dollar company (Saas, etc) can be run with very sane and simple stack: - 1x web-server [$FOSS_or_COSS, iis] # COSS => commerical open source e.g caddy. nginx is both - 1x application-server [.net, java, python, php, and all other langs. heck PERL too!] - 1x database-server [$FOSS_or_COSS, sql-server, oracle, etc] Virtualized or baremetal. Containers not necessary! Increase it by 2 or 3 for HA if you so desire.

Engineering teams should aspire to advocate "We run this entire thing on these few resources vs our application runs on 26k nodes" The latter just means we've lost the plot and throwing money to solve a problem. Forgive my sarcasm here, but well done k8s&&containers!

Don't see any reason why FB can't run on the same comparative hardware as SO. I.e. replace the CCSS (commercial closed source software) bits with the FOSS/COSS ones. And when we take it in as a whole, this is much more greener/sustainable to work with.

Also, there is a legend out there that I heard of many moons ago :) ... THERE is a billion dollar company running on a single sql-server and single iis web-server

If you read further in the article there appear to be 6 SQL Server machines (across 2 clusters). Those are some highly spec'd servers with huge amounts of RAM and CPUs. With decent caching the SQL Server machines wouldn't be hit too often either, hence the Redis and Elasticsearch servers.

Fewer, but larger DB servers would typically mean much less maintenance and dev ops headaches.

As others have said, skipping best practices is typically reasoned out really well before proceeding. Not everyone needs every best practice.

If they'd used open-source software rather than the Microsoft stack, they could have scaled horizontally without having to worry about buying more software licenses.

Personally I'd rather manage a small number of large servers, compared to managing a large number of smaller ones.

The licensing costs associated with Stack Overflow also can't been that large. Sure Microsoft software isn't cheap, but it's also not that expensive. The licensing cost is easily offset by staying on a technology stack their employees know.

> The licensing cost is easily offset by staying on a technology stack their employees know.

Stack Overflow wasn't developed by an established software company. It was a completely new startup, founded by two high-profile (at the time) software commentators.

They could have hired people to use any stack they wanted. While the state of web development wasn't what it is today, OSS stacks such as the LAMP stack were fairly well-understood and popular.

That's a reasonable point, but still is the cost of the licens higher than the cost of one or two additional developers? Jeff Atwood was/is if I recall a pretty talented C# developer, so why onboard extra people. It's more than just their salary, there's a cost beyond that associated with hiring people.

I'm not arguing that the licens isn't a concern, I just question how much of a concern it really is, compared to everything else.

The MS stack is affordable to developers working on developing countries and free for small products (with a lot of strings attached).

Of course, the OSS stack costs nothing to acquire, is cheaper to admin, and will give you more functionality and easiness of use, but the MS one isn't some unaffordable monster like other proprietary ones.

Stack Overflow existed for a couple of years before taking any funding and it was developed by three people who were friends (Jeff Atwood, Jarrod Dixon and Geoff Dalgas) none of which were even getting paid regularly at the start and who were all primarily familiar with the .NET stack from previous jobs.

It makes total sense why the tech is what it is. One could argue they could have shifted away from that stack since then, but one could also easily argue (very convincingly to anyone who has ever been involved with any significant project rewrite) "but... why?".

> The licensing costs associated with Stack Overflow also can't been that large

Because we went with the reliability of a full Microsoft stack—.NET, C#, and MSSQL—our costs grew with the number of instances. Each server required a new license

They literally mention license costs as an architecture driving constraint.

It's probably only costs for SQL Server, because anything else (ASP.NET, .NET, C#, etc.) is free and open source.

You couldn't run ASP.NET in Linux back then, so you had to pay for a Windows license, too.

Stack Overflow predates any of those being open source.

Wouldn't they have to pay for IIS as well? (Not mentioning Windows itself?)

Windows server yes, you pay for that, but IIS comes bundled.

Sorry, I wasn't making that clear. It's not that it isn't a factor, they clearly think it is. The question is how big a factor is it really? My experience is that the licens is less of an issue, compared to the time and resources you'd otherwise need to redesign everything.

When Jeff Atwood and Joel Spolsky started Stack Overflow they had a podcast. In that podcast Joel repeatably argued that hardware wasn't that expensive, compared to Jeffs time. His argument was that buying a larger server and scaling that way was much more cost-effective. I think that's still true, to a point, even if you didn't have the licensing cost.

> [...] buying a larger server and scaling that way was much more cost-effective. I think that's still true, to a point, even if you didn't have the licensing cost.

I don't know if in the cloud world this still holds true, considering that (at least in AWS) if you vertical scale 2x you get usually a 2x price increase. Clearly I'm not factoring costs of making an app horizontally scalable or DB-shards aware (which I think was always the main selling point for SO vertical scaling: they usually referred to the DB, and they were against sharding)

> Because we went with the reliability of a full Microsoft stack

As opposed to what? What reliability are they worried about trading off?

Don't get me wrong - I love C# and .NET, and MSSQL is a fantastic RDBMS. And if your people already are familiar with this tech stack (which I think is the case here), it makes sense to play to strengths.

But let's not pretend that this stack is the more reliable option to any other alternative...

Stack Overflow started in 2008, what exactly were performant alternatives back then? Java EE + Oracle/Postgres?

Cannot comment on reliability, but considering .NET and MSSQL are from the same company, I'd assume they work great together.

That was just a few years after Postgres became competitive.

Postgres was already a better choice than Oracle or SQL Server on almost every way (the clustering setup UX sucked). But very few people were aware of that, and they would have to learn a completely new stack to use it.

I don't know if SO is using everything that comes with MSSQL, but it's a lot more than just a RDBMS. SSIS for example is a full workflow/job execution environment fully integrated with MSSQL. When I did a similar analysis to move off of MSSQL in the past, there is a lot more to it than just swapping out databases.

In our case we were also using MSAS and SSRS.

SSIS, MSAS and SSRS are minimally functional products that exist from stopping MS customers from looking elsewhere. If they are adding value to you (instead of just being demanded from the high up and never used), it's better to look at the alternatives, some times even if you are already paying for them (that one is a difficult decision, as the others are kinda expensive).

Anyway, the Microsoft implementation of data workflows is so stupid that you are better of with manual dblinks or creating the entire thing in a general purpose language. Their largest competitors share a lot of the same problems, so I'd say that workflow software is just an excuse to hire low-paid professionals and a real drag on any working development team.

I worked at the MySpace parent company during the whole growth explosion. Microsoft would send us custom MSSQL patches to fix issues the team was running into. Sure, someone could have patched an open source database but not everyone wants or needs to staff engineers of every category.

Software licenses are usually peanuts versus developer salaries and overall infrastructure costs of the whole company.

Agreed. It's always the people not paying for things that seem to be complaining about paying for things.

Depends on where you live. Certainly not peanuts here.

Edit:

I know that if you spent your whole life in California it can be hard to believe that in some places junior developers make $250 a month (senior more than that, but typically less than $1k a month), so a single Windows Server license buys you four months of work of a single junior dev (however unproductive it may be), or 1-1.5 months of a senior dev's work.

In my current place of work a few years back I started a strong push towards Linux and as much FOSS as possible, and over some time (as new projects were written and some legacy stuff re-written) it resulted in a massive reduction of total operation costs.

I didn't expect any other reaction, though. You understand our realities better than us.

They also commented that the licenses paid for themselves in reduced hw spend, as AOT compiled code plus very performant SQL Server allowed them to run in fraction of the size comparable sites operated.

Would 10x the amount of hardware be a problem for them in any way?

Did they aggressively optimize their setup because of hardware costs?

they were losing money in the first few years. struggling to find way to monetize it. at one point were selling programming books with amazon affiliation.

these types of cost cuts make the difference between life and death in early stage startups.

i believe they still do that

Reply on news.ycombinator.com