Nick-Craver

Perhaps I can add some context on top of the post. All updates are in transactions unless explicitly opted out. Also remember our case is a little special: we have hundreds of the databases with the same schema, what if #328 fails? The failure/rollback scenario is a bit more complicated when you go past a single database involved.

As for backups: absolutely. We handle this independently though. We do full backups every night as well as T-logs every 15 minutes. If I had to restore every database we had to a very specific point in time or just before a migration command was run: we have T-logs to do that going back 4 days at all times.

I'm sure there are good solutions for single database applications way more fully featured than our approach, they just do little to solve any problems we actually run into.

The problem here is that these services move - so if it's in /etc/hosts, our failover mechanisms (to a DR data center which has a replica server) are severely hindered. We're adding some local cache, but there are some nasty gotchas with subnet-local ordering on resolution. By this I mean, New York resolves the local /16 first, and Denver resolve's its local /16...instead BIND doesn't care (by default) and likes to auth against let's say: the London office. Good times!

This, more than anything.

There's no compelling reason to invest in such a move for our situation. We're very interested in how Open Compute progresses, though.

These plans are in flux. At a core level (heh), we nee to have libraries ported first anyway. So we're doing that now. Jil, StackExchange.Redis, Dapper, and Sigil are up on NuGet and core compatible today (some as pre-release). Others like Exceptional are just pending some RC2 goodness to release, and I'll be starting on MiniProfiler after that. We'll likely post other internal tools to .Net Core and start heavy testing in the RC2 time-frame.

Will we run Stack Overflow on core? Probably not for a while. I was asked exactly this in a recent On.Net interview at the 24:22 mark; you can listen here for reasoning: https://youtu.be/DJn8-Psznsw?t=24m22s

It's at 1.5TB of data and 200GB of T-Logs these days.

I commented a bit here with the architecture post: https://www.reddit.com/r/programming/comments/468p2m/stack_o...

But a better answer is: that price keeps changing. We now have a lot of AWS and on-prem experience in house to do a great post. We'll be doing a lot of research and proper comparisons as a huge part of that upcoming post: https://trello.com/c/4e6TOnA7/87-on-prem-vs-aws-azure-etc-wh...

To clarify a bit: I meant they are non-essential to serving the sites. If they all went offline, visitors to Stack Overflow would not be aware.

In the Stack Overflow cluster, the RAID 0 NVMe contains Stack Overflow database and the SATA SSD RAID 10 holds the others (Mobile, Translations, PRIZM, and Sites).

In the Stack Exchange cluster, the RAID 0 NVMe array contains all databases except for a large log database (which I called Careers.BigStuff, because it seemed like a good idea at the time). This larger log database is much more rarely accessed and is on the 10K HDD RAID10.

We run backups on the primary for several reasons, but they are all sent off-box. We have 2 primary on-site backup servers, then those backups go offsite and to tape. Database backups are every 15 minutes with T-Logs and full backups nightly. We also run copy-only backups in the DR data center nightly as an additional backup measure.

That's coming up! Deployment is the next post in queue: https://trello.com/c/bh4GZ30c/25-deployment

Ah I see the confusion now. To clarify: we use redis for a great many things, not just serving HTTP requests. For example, we use redis lists and sets to continually recalculate mobile feeds for users - that's roughly 4 billion of the hits every day.

We don't really cache that much, for example: every post, comment, user, etc. on the page is pulled and rendered live. We output cache certain pages for 60 seconds for anonymous users, but it has almost no performance impact (I think it's a 4% hit rate?) Some things are cached, but each page render is very dynamic - more so than most I think? I don't have a great source of comparison for similar sites though.

I wrote this post to make as many developers and sysadmins aware of the (very critical) bug in .Net 4.6 as possible. Please take a minute to read and make anyone deploying (or thinking about deploying) .Net 4.6 aware of the issue. If you have any questions, I'm happy to answer here, on the blog, or on reddit.

This isn't true anymore. Octopress uses an older version of Jekyll. If you're on 3.0 (which we're using) has incremental generation. Here's the log of a recent incremental build:

  [Step 1/3] Jekyll build (11s)
  [20:52:40][Jekyll build] Jekyll build --config '_blog.config.yml' --no-watch
  [20:52:43][Jekyll build] Configuration file: C:/BuildAgent/work/4e27a39603eb2bf1/blog/_config.yml
  [20:52:43][Jekyll build]             Source: C:/BuildAgent/work/4e27a39603eb2bf1/blog
  [20:52:43][Jekyll build]        Destination: C:/BuildAgent/work/4e27a39603eb2bf1/blog/_site
  [20:52:43][Jekyll build]  Incremental build: disabled
  [20:52:43][Jekyll build]       Generating... 
  [20:52:52][Jekyll build]                     done in 9.082 seconds.

This is a photo of the sales area. Developers are downstairs and have their own offices. If you want to browse on streetview, it's here: https://www.google.com/maps/@40.708443,-74.007121,3a,75y,234...

Source: I'm Nick Craver, a Sysadmin and Developer for Stack Exchange.

As Kyle says, we were helping our sister company Fog Creek keep their servers online (as well as other people in that facility like Squarespace) because we cared. Our traffic was not being served from that data center and in fact we shut down most of our servers during that to conserve generator fuel. Our traffic was flowing just fine from Oregon. A decision Kyle and I made the night before when concluding they would probably shut down power to lower manhattan in preparation for flooding.

When your neighbor's house is on fire you don't argue over the price of the hose. You help. Our remote people that couldn't come help in person also helped them replicate their entire network in AWS as a backup plan.

I don't usually post pissed off comments, but you're dead wrong here and intetionally or not demeaning a good company and good people whom, because they cared, came to help in a time of emergency. I take it you weren't in New York during Sandy; it looked like a post-apocalyptic war zone afterwards.

TL;DR - You don't know what you're talking about.

I can't resist reminding you that 1 command from a sysadmin routing traffic to the wrong network was the cause of the last major outage there :)

They are getting much better, as all providers are. They're still just not a fit for many people because of performance requirements that are either impossible or too costly to meet on that type of infrastructure.

I've always said this: the cloud isn't a good fit for us; do what works for you.

It normally would, except the base IIS page is a proper 200 to all requests. The load balancer removes all servers having issues - from HAProxy's point of view things were dandy.

As always, we are looking how to adjust our deployment process to prevent it from happening in the future.

I haven't had nearly as much experience as George and Shane have with them after initial spin up. I'll get one of those guys to chime in here.

Thanks! We honestly do this for us and my mild cabling OCD, but if we can help anyone by sharing the details then it's worth it.

The cloud is awesome if you have zero interest whatsoever in hardware. It's not without trade-offs (nor is the other direction), and too many for us - but if it works for you then great I say.

We obviously feel very different, and are just doing what works best for us.

We have a hot slave at all times for all instances. In the event of a failure the second slave will kick in. All applications are already connected to both servers via the StackExchange.Redis (https://github.com/StackExchange/StackExchange.Redis) library. The mechanisms for failover, etc. are built in there. We use this library via the dashboard (pictured at the bottom of the post) in Opserver (https://github.com/opserver/Opserver) to do quick swapping of master/slave, etc. We can do this during the day without anyone noticing.

Slaves are not just for backups - they also serve as pub/sub mechanisms. Every publish propagates to slaves. Since we use pub/sub for things like web sockets, we can easily move that entire concurrent connection load to another data center with a simple DNS change. Yep, we've tested this - it worked well.

It is of course noting: redis just doesn't fail. We had one out of memory fail for the Q&A web sites when forking and that's it. It has been rock solid here.

We've only gotten 2 that I can remember in 4 years here. One was a regression in .Net 4.0 with high load hashtable inserts (which was previously fixed in 3.5 SP1) that they had to re-fix. The other was a SQL server issue in 2014 CTP1 we were running with an availability groups edge case on high volume/high latency secondaries.

Unless we're testing beta software, the MS stuff generally works and works very well. We of course work with them on feature requests that will make life better - some happen, some don't. I'm on the opinion you should try and have this relationship with any vendor or open source project. We're trying to make what we use better, and not for just us.

Nick Craver - Stack Exchange SRE

To clarify (and I'll get the page updated to make this clearer) we're handling 185 requests/sec per server on average. There are other things not even touching IIS of course, so we're usually seeing somewhere between 2,400 and 3,000 requests per second overall coming into the network (not including WebSockets for various reasons).

Here's a grab from bosun taken just now of web tier load on those 9 servers: http://i.imgur.com/RJeEiYV.png

And here's a view of overall traffic coming into the NYC data center (except WebSockets): http://i.imgur.com/LV30989.png

Nick Craver Stack Exchange Developer & SRE

Yes, we'll move to vNext when it's a bit more stable (likely before RTM). We're already compiling with Roslyn for localization for example.

This is simply not true - it assumes that everyone's forwarding server for an unknown record is the root servers directly, which isn't the case.

If this were true, my life would be much easier because DNS propagation would be much more predictable than it is today. One bad server in the chain serving obnoxiously high and invalid TTLs can ruin your day. It's not very common percentage-wise, but it certainly happens every time we switch DNS over for Stack Overflow.

I'm not sure what you're comparing here...Linux is a registered trademark, Markdown is not. There really is a big distinction here, so comparing them as equals doesn't work.

Is Markdown actually trademarked? It's beed used in similar fashion for years and I've never seen a trademark referred to anywhere.

You can't copyright a name, which seems to be his main point of contention judging by the public responses he has offered.

This has nothing to do with my view point, it's simply a fact of law: you can't copyright a name. No matter what text you put somewhere says, it doesn't magically apply.

It depends on your definitions of "a lot" is. It can scale pretty high by default, but when you're doing hundreds of millions of objects in a short window, you can actually measure pauses in the app domain while GC runs. These have a pretty decent impact on requests that hit up against that block.

The vast majority of programmers need not worry about this, but if you're putting millions of objects through a single app domain in a short time, then it's a concern you should be aware of. This applies to pretty much any managed language.

Hardware is cheaper than developers and efficient code. But you're only as fast as your slowest bottleneck and all the current cloud solutions have fundamental performance or capacity limits we run into.

Could you do it well if building for the cloud from day one? Yeah sure, I think so. Could you consistency render all your pages performing several up to date queries and cache fetches across that cloud network you don't control and getting sub 50ms render times? Well that's another matter, and unless you're talking about substantially higher cost (at least 3-4x), the answer is no - it's still more economical for us to host our own servers.

And yes I realize that's a fun claim to make without detail. As soon as I get a free day (I think there's one scheduled in 2017 somewhere), I'll get with George to do another estimate of what AWS and Azure would cost and what bottlenecks they each have.

Nick Craver - Stack Exchange Sysadmin & Developer