How Stack Overflow upgraded from Windows Server 2012
stackoverflow.blog
stackoverflow.blog
What is the Linux equivalent of this operation? apt-get dist-upgrade? Now that Windows 10 onward is rolling release (is it?) is this the last time SE staff is going to have to go through this?
This caught my attention:
> Putting the 2019 server into the existing 2012 cluster – as expected, this fails due to the operating systems being different.
How can a website be this tightly coupled with the operating system it's running on? Shouldn't Microsoft be blamed that their operating systems are not backwards-compatible? To my knowledge, Microsoft goes to extreme lengths to assure backwards-compatibility of the userland apps. What went wrong here?
I have upgraded a website from Windows 2003, to Windows 2008, to Windows 2012 and now it is running on Windows 2019. I jumped 2014 and 2016 because upgrades weren't necessary for my needs. Each time, I only spent 1 hour maximum to migrate the system. I am not even a server guy.
On a production cluster, you aren't just going to "apt dist-upgrade" and be done, you're going to need to deal with versions of all sorts of things under the hood changing versions as well. In the SO case, they were at least able to keep the database versions the same, but it sounds like some other things were incompatible between the two OSes that prevented them from directly clustering together.
Over the last couple decades, I've done something around 100 of these sorts of upgrades for clients of a managed hosting service I used to run, and they were rarely simple OS upgrades. Lots of planning and testing went into them.
That's the sort of upgrade this article seems to be about, I'm not sure if it's because SQL Server is tightly integrated with some OS-provided services that were incompatible, but it was certainly more involved than a base OS running only the database and just installing the same database on the new OS.
The post is about their database servers, not the site. Microsoft SQL Server’s high availability and disaster recovery solutions are built atop Windows Server Failover Clustering, which comes with a lot of gotchas. For example, for over a decade, all of the members of a cluster had to be the same major version of Windows. That made upgrades not just difficult, but impossible: you had to build a brand new cluster from scratch every time and then migrate over to it, and destroy the old cluster when you were done.
Microsoft does indeed go to lengths to assure backwards compatibility. This isn’t a compatibility issue - it’s an upgrade process issue.
(Source: I did SQL Server HA/DR work for years, including helping Stack out along the way, but abandoned HA/DR work when it became kinda untenable. Now I focus on the performance tuning side instead.)
Non-rolling linux releases (such as ubuntu) run into all sorts of strange issues during a dist upgrade, and you may end up with database servers sufficiently different to prevent them from clustering together.
Windows server has always been versioned differently from desktop windows with 8 LTS versions since 2003.
I've been a software engineer for over two decades now and I've seen stuff come and go. The products Microsoft is shipping today are impressive bits of technology and, and the end of the day, do what you want them to do and do that very well. And I've been through the more difficult years where I wish I had thrown SQL Server off a cliff.
Why am I bothering to write this? Because I'm still in technology and I wish people would stop jumping on the next bandwagon or pointing to the next shiny object. There are tried-and-true technologies out there that shouldn't be ignored based on prejudices. Stack Overflow is a great example of a company who understands this.
https://stackexchange.com/performance
More details are scattered across various places, e.g. Nick Craver's blog (https://nickcraver.com/blog/) and the Stack Overflow and Stack Exchange meta sites.
This really underlines it. Running one of the most popular sites on the web with just 9 web servers, 2 SQL servers and a smattering of dedicated role servers. But that popularity ultimately boils down to 450 req/s, and they've got a nice, fast site.
No faffing around with "scaling" up and down to handle the fact the cloud servers are slow. It always feels you get nowhere near the listed performance and are constantly getting throttled.
I would guess a lot of startups would be better off paying a few grand a year for a couple of dedicated servers and just forgetting about AWS/Google Cloud/Azure.
Way higher than that. Was 2X higher back in 2016, even: https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar...
Second, it also depends on wether you are write or read heavy. SO is primarily read heavy and they can get away with most requests never hitting their main servers but instead being handled by a CDN.
But overall I do agree. Dedicated would give you a performance boost compared to virtual machines. And sometimes it might be worth it to bring stuff in house and do it yourself.
Is HTML a monopoly? Is JS a monopoly?
Because the next best way to change a standard is add another one?
For all the moaning about javascript over the past decades, it has evolved into a fairly formidable language since 2015.
I’m not even sure introducing a new language would help anything at all. It would simply be quickly adopted by those the very same abusers.
Maybe the solution instead is to have better browsers and implement better privacy policies.
However there is a large amount of diversity within Linux, so I would hesitate to call it a monoculture.
They had already got experience with it! That's why they chose it. In my opinion go with the stack you have experience with, don't just do what everyone else is doing.
https://www.tarynpivots.com/post/how-stack-overflow-upgraded...
That statement is not entirely accurate. The SSDs certainly did die, but not _because_ of the 2019 upgrade, it just happened at the same time as the 2019 upgrades.
The major 2019 SSD issue was however that Intel and Dell kept passing the buck on who was to blame for not having 2019 drivers for the NVMe SSDs. Eventually I ended up having to bully Intel into giving us the drivers. We decided that "bullying Intel" was not an appropriate future tactic for getting the next version of the drivers, so that's another factor for staying on 2016 (which has drivers we've tested for years)
All in all, well under $50k in licensing costs. Basically a rounding error, if you compare to something like AWS billing for a website of the same scale.
I wish. Microsoft SQL Server Enterprise Edition (which is required if you want to use over 128GB RAM for caching data, or servers with over 24 cores) costs approx $7,000 per CPU core.
Our websocket process currently has ~500k concurrent connections across 9 web servers.
Nick has some information on his blog (https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar...) although it’s a little outdated - we no longer use Netgain, we use the .NET Core websockets bits instead.
https://www.tarynpivots.com/post/how-stack-overflow-upgraded...
I worked as a .NET dev for nine years, and although I've never had to deal with upgrades at this level before, I can appreciate that at the kind of scale that Stack Overflow operate, and considering they run on physical boxes (a rarity in any stack nowadays) any kind of upgrade between major database and OS versions is going to result in its own issues.
While I don't work on the Microsoft stack any more, I've always appreciated the way that Stack Overflow build their software. Their engineers and DBA's seem to be a who's who of talent, and they both build great services and put out great content, at scale. It's doubly-surprising when so many people used to give Jeff Atwood so much shit for not delivering anything of note when Coding Horror was one of the top programming blogs on the web.
The stack might be very different to anything done in 2019, but the fact that I can't think of many times where Stack Overflow went down shows that they're doing a great job, and hopefully they'll continue to document the journey for years to come.
It was quite the coup at the time for Microsoft, especially the SQL Server team, to have such a high profile, high-traffic site.
Additionally, SQL Server is a very good database, fast, reliable and feature-rich.
Additionally part 2: Microsoft owns Github, can’t get more open source than that!
> Spolsky is former MS and familiar with their stack
That is key: often experience (i.e. the cost of problems caused by lack of experience) can outweigh licensing considerations, particularly in the fast moving PoC/prototype stages and the subsequent quick growth phase.
> Additionally, SQL Server is a very good database
No argument there. While other DBs could certainly do the job too, SQL server is a good choice, even without that "experience bonus".
License costs wouldn’t be a good reason to pick another stack over the one you know.
Although, tooling was very heavily Windows based at the time (no Rider); so it is understandable why StackOverflow took the direction they took.
It was an amazing achievement, but never quite got me there.
Source: Was runnin .NET on Mono 7 years ago or so.
That's not a source. This is a source:
http://www.techempower.com/benchmarks/
Yes, it does show Mono's ASP performance is substantially worse than .NET, and probably more-so in 2008. I never said otherwise.
There are a lot of things to take into consideration when selecting a tech stack for a brand new, entirely unproven product. It's certainly on the list, however if performance ranks highly, then I'd be seriously questioning the diligence of whoever made said list. Expenditure (e.g. Windows Server licensing fees) would rank high on the list, however so would developer familiarity with a given stack as well as tooling familiarity.
As I said, I can see why StackOverflow made the decision they made, but I'm doubtful performance was, at that time, a major driving force.
At the time, major "web development" stacks that people would have thought of on Linux tended to be much more resource hungry (still are), meanwhile the people behind StackOverflow had very good .NET skills that they could apply immediately. This translated into being able to serve more people per server than if they used, let's say, PHP, Ruby, Python, etc.
And as people said, SQL Server. Which is a very, very capable database server with impressive performance.
Not sure how the story goes for Rails, but Python (Django) used to be a pig when SO was started over a decade ago, for similar reasons as PHP.
Django's execution model is not at all similar to how PHP code is being run. Pretty much every python web server uses wsgi (or mod_python before that) which relies on a persistent app server. Whatever are the reasons for django's assumed performance issues, PHP's runtime model had nothing to do with it.
SQL Server has huge amounts of tuning and cooperation with NTFS driver team to ensure good performance, and frankly, it's at the high-end of the databases, and I really wouldn't mind using it except for price :(
C#.NET performance is up there with C++. C#.NET is only slower than C++ during garbage collection. When garbage collection is done, it is back in business and running at the same speed.
That's not a great summary. .Net is complex. There's AOT compilation but not available for all purposes. There's JIT which applies most of the time to C# code compiled into msil. From there different things can happen depending on implementation, which may also include running the code via an IL interpreter.
Put those things together, and it’s fairly evident that C#/.Net greatly outperformed the Django/Rails stacks that were popular at the time.
2. JavaEE??? What has the StackOverflow team done to make you hate them so much? :o)
More context: https://blog.codinghorror.com/why-ruby/
But I guess that is something people will argue over for a long long time.
Well... my opinion, based on my experience...
Anyway...
I was certainly not saying that there are not scenarios where it is better to go with windows servers. Things don't happen in a vacuum.
This isn't true. I managed and supported a fleet of Windows and Linux servers for a web hosting company for 14 years. Both OS's have their time sinks and gravity wells of failure.
You can disagree with me ok, but if you want to convince me that I am wrong it would be nice with substantiated material.
As a sysadmin with 30 years of experience running Linux and Windows servers though, I'll offer my own anecdote - I'd say that Windows is far easier to manage for many common scenarios. It's all point and click and you can basically figure out what to do without looking at any documentation anywhere.
It's so easy, you can teach a monkey how to configure a workgroup, domain, users, dns server, firewall, etc. on Windows.
Also, which Linux distro are you talking about? What with all the fragmentation, nothing is ever the same on every distro.
Figuring stuff out without documentation when there is documentation is a bad idea. If you know what to do the GUI can guide you a bit, but it does not do work for you. If you want to do something that you don't understand you should really read more about it, the context being irrelevant.
I doubt you could teach a monkey to do anything like that.
At my last work we used the same distro for everything, why make it hard for yourself. Personally I think most things are the same in the most commonly used distros. Everything is generally just a file. Use mount to mount disks /dev /etc /var folders all standard normal stuff.
You're ignoring overall Total Cost of Ownership (TCO). Licenses are a drop in the ocean compared to the cost of long term ongoing management of your platform(s).
Wether they choose to move to linux is a different matter.
(Windows probably makes it a bit hard though)