Hacker News Reader

Comment by ed_elliott_asc | Hacker News Reader

I get a bit annoyed by this sort of post, blanket statements like:

"With our code base, you won’t be able to easily do test driven development or similar practices that the industry seems to love."

Why does the industry seem to love testing? Does the industry love testing? Does this mean you do or don't love testing?

It also feels like they wrote this blog post to show off: we write our c# code as if it was c code because we are better than all of you.

The fact is that every application is different, they chose to use a language that back in 2000's wasn't designed for performance and so they paid the price of using it for performance and didn't want to pay the scale out price.

Is their codebase testable, yes completely. Will adding unit tests to their code destroy their performance? no, there are always ways to make it testable and fast. Maybe in a few years we will see a "We thought testing was impossible but we made it work and also keep our codebase fast".

View on HN

They're really not selling their codebase or practices to me to be honest. If their system is built like that it might have enough performance to run a top x site on only a handful of bare metal servers, but the article also implies there's a lot of "here be dragons" code - don't touch this, you are not clever enough to test this, and since we don't have tests, you might break it by touching it.

Which is just bad practice on the one hand, and elitist on the other.

I for one am thankful I don't have to work with those constraints, and I can adopt a better mindset - make it work, make it pretty, make it fast, for example. I think you cannot optimize a system if you cannot read and understand it.

Also note I explicitly said "system", not "codebase"; at a big enough scale, you can't look at performance as a factor of code, but of architecture and patterns. You need to be able to get figures from specific systems in the whole architecture, not individual lines of code. Make the architecture easy to analyze and optimize.

> the article also implies there's a lot of "here be dragons" code

Agreed, fine if enough of the core developers are around, but if the company merged, the app needed to be extended or altered significantly, or some of the underlying technology stack is deprecated/becomes unsupported keeping things working will be very expensive.

Reminds me of "tech debt as a lack of understanding"

https://daverupert.com/2020/11/technical-debt-as-a-lack-of-u...

> If there’s been enough organizational turnover or enough feature creep in a product, then maybe doing a rewrite is the best option so your team has a collective understanding of the code. You can’t expect people to be productive in something that was a culmination of [complicated "here be dragons" code] made by people who no longer work there. At that point your technical debt balloon has popped, you are in possession of a toxic asset.

I think that they are in the situation where it can't be altered significantly which ironically is probably a good thing for stack overflow - they don't have reams of features.

> Which is just bad practice on the one hand, and elitist on the other.

I got exactly the same impression. Making a strawman out of the rest of the market and special-casing yourself always sets off a red flag. It indicates the possibility of a culture could be out of touch with reality, that chooses to rationalize its poor decision rather than facing reality. I wouldn't work at a firm with this kind of culture, it's a liability.

I once worked in a shop where a few teams made blanket rules for clubs of reviewers for different core technologies. There was a Python reviewers group a Puppet reviewers group etc.

The membership of these groups bore little resemblance on the technologies people used and had to work in day to day, but instead reflecting the current group politics of a large engineering organization. As the reviewer enforcement was wired into the deployment tool - appeasing these reviewers was required to be productive. In practice the reviewers would road block entire team's initiatives.

The end result was a proliferation of technologies across the company such that teams could practice team code review rather than adhere to the wims of the anointed reviewers.

I also found the "or similar practices that the industry seems to love" coming across as a bit vague and even arrogant.

On the other hand, I think that they have earned bragging rights - we're talking about a site that has massive scale and reach, and has a reputation for being a very efficient system. I appreciate that people like them share their views, even if they are contrarian and a touch arrogant.

> they chose to use a language that back in 2000's wasn't designed for performance and so they paid the price of using it for performance and didn't want to pay the scale out price.

In terms of more performant languages at the time what could have they picked? C++? It sounds like they would've lost a lot of the conveniences that a framework like .net gave them. Isn't the fact that they avoided paying the scale out price remarkable?

It's important to remember Joel Spolsky, one of the founders of stackoverflow is an old school Microsoft guy that managed the Excel team before starting his own company. He has always been a bit of a Microsoft fanboy, and used to advocate a lot for VB as a serious language back before it got swallowed by .Net and turned into a less powerful syntax for C#. The fact they chose C# isn't surprising at all given that. It actually would have been far more surprising to see them pick anything but C#.

I think the previous poster was a bit off the mark though. The language performance wasn't really the issue there, rather it's the fact that they picked a language that at the time really only ran on Windows, and as a consequence they were forced into running their web servers on Windows. That choice then forces them to scale up rather than out since each instance has license costs attached to it. For most companies running on Linux, it's trivial to scale out since your only costs are the compute cost (or the hardware cost in a non-cloud model), where as it tends to be far more expensive to scale up as more powerful hardware tends more towards geometric increases rather than linear. These days the choice of C# wouldn't be such a big issue as .Net core can easily run on Linux servers, but back in the 2000s using C# was putting a pretty big albatross around your neck by way of Windows licenses.

> it's the fact that they picked a language that at the time really only ran on Windows, and as a consequence they were forced into running their web servers on Windows.

If the founders and early employees were from Microsoft it might have been easier for them to use Windows Server since they were already pretty well versed in Windows development.

It's a pattern I constantly see: "Why did your startup use X instead of Y?"

"Ohh well X has this feature that Y lacks and so and so... ohh and the founder and his friend were pretty good at X and used it before."

Yea they mention that in the post.

I agree with you that it seems like a self-imposed limitation. At the same time, it makes one think how said limitations can actually foster creativity and efficiency. They mention in the post - they constantly gloat about this - that they could run SO on one or two machines.

I'd imagine that said machine would need to be a behemoth and not a t3.micro, but intuitively I feel that this would be much cheaper than the average horizontally scaled web application. Or in other words, that they're hyperefficient, regardless of architecture.

Does anyone have any insight on whether this intuition is on the right track?

Eh, not sure that follows. Here's the thing, costs aren't linear. If you do what AWS does and create some artificial "compute units" as a sort of fungible measure of processing power, what you'll find is that the sweet spot for price per compute unit is a medium power system. The current mid-range processors tend to be slightly more expensive than the low-end processors, but significantly cheaper than the high-end processors.

So, hypothetically, lets say you can get one of 4 processors, a low end one that gives you 75 units for $80, a mid-range processor that gives you 100 units for $100, a high-end processor that gives you 125 units for $150, and the top of the line processor that gives you 150 units for $300. If you normalize those costs, your 4 processors get price-per-compute values of $1.01, $1, $1.20, and $2. The best value is at the $1 per compute unit price point of the $100 processor. Logically if you need 150 units of compute power you have 2 choices, you can use 2 $100 processors, or 1 $300 processor. Clearly the better option is the 2 $100 processor. This would be scaling out. In the case of what SO did though, they took that off the table, because their formula isn't just the cost of the processor (ignoring related things like RAM and storage), but also includes a per-instance license cost. Their math ends up looking more like $100 CPU + $150 windows license times 2 totaling to $500, vs, $300 CPU + $150 windows license times 1, totaling to $450, which ends up making the more expensive processor the cheaper option in terms of total costs.

> If you do what AWS does and create some artificial "compute units" as a sort of fungible measure of processing power, what you'll find is that the sweet spot for price per compute unit is a medium power system.

At least when I look here the costs are linear.

E.g. c5.2xlarge is double a c5.xlarge, and a c5.24xlarge is 24 times higher than a c5.xlarge.

https://aws.amazon.com/ec2/pricing/on-demand/

That's interesting, it didn't used to be linear, particularly on the very large instances. Oddly when looking at the prices for RHEL those are much closer to what all the prices used to look like (I hadn't actually looked at AWS pricing in a few years). I wonder if AWSes virtualization tech has just reached the point now where all processing is effectively fungible and it's all really just executing on clusters of mid-range CPUs no matter how large your virtual server is.

They were/are on bare metal so they have real dedicated cores and no VM overhead. Two bare metal cores is a hell of a lot faster than two virtual cores. I had been away from infrastructure during the rise of vmWare and didn't realize that the core count was potentially fake. I was sparring with an admin over resources one day and mentioned the number of cores on the VM and he just laughed. "Those are virtual, you don't really have four cores." Be careful in your assumptions about these VMs in the cloud.

He never managed the Excel team and he didn't write code for the product. He was a PM and wrote the spec for VBA. An accomplishment to be sure but not much different than all the other great individual contributors Excel has had over the years.

Perhaps I should have said he was a manager on the Excel team.

He did not have a management role. PMs write specs and talk to the customers. That time was also very much the "cowboy coding" era so part of the job was convincing devs they should implement it how he spec'd because he couldn't force them. I think that's part of why he was so popular with his blog aimed at programmers, he had lots of time honing his persuasive technique.

> It sounds like they would've lost a lot of the conveniences that a framework like .net gave them.

Would they have? It sounds like they put a lot of effort into bypassing the garbage collector by creating buffers up front and then never letting then reusing them so they can never be garbage collected. Best practice in embedded C code is to allocate buffers up front and then reuse them. The terminology in the above two sentences is different, but the way the code is written is exactly the same.

Once you lose garbage collection does .net give you anything over C++? I don't know enough about .net to answer the question - even if I did the answer probably depending on if you are doing something where .net has a saner syntax, or run of the mill code that looks almost identical in C++ as .net (C++ templates are legendarily bad for a reason, if .net is even slightly better it doesn't take too many generic things to make .net better)

> Once you lose garbage collection does .net give you anything over C++?

You don't need to lose garbage collection altogether. There's a middle ground here: if you can switch to manual buffer management for the most GC-heavy, speed-critical code, that still lets you use GC with impunity everywhere else.

> I don't know enough about .net to answer the question - even if I did the answer probably depending on if you are doing something where .net has a saner syntax, or run of the mill code that looks almost identical in C++ as .net

I don't know enough about either C#/.net or C++ - I was merely speculating.

I know there are some web frameworks for C++, but I've never heard of any big websites being built with them. I am assuming that, even back then, .net provided a lot of modules for building websites, not only templating but also auth, email, managing statics, an ORM (I know they ended up rolling out their own, but maybe they used .net's in the very early stage), etc.

Pretty sure Amazons entire backend is C++. Hard to get much bigger than that.

they seem to love it...non-committal.

Adding test coverage might indeed destroy their performance, in that they might expose where their system's bottlenecks are and realize that they've been focusing on the wrong things. When it comes to performance optimizations, folks' intuitions are almost always wrong. It's important to actually test and measure stuff like that.

I think it's true in general that modern CPUs are insanely fast at executing code within a single process, and context switches are orders of magnitude slower due to their overhead. So, monolithic systems are going to be intrinsically "fast" compared to systems that pass messages through an operating system. They seemingly made a good decision for their project there, whether or not they made that decision rationally.

It's possible they invested a lot in infrastructure for runtime performance profiling and monitoring and therefore know exactly what pieces of code to optimize. Reading the article, though, they're seemingly sweating the overhead of polymorphic function calls. The overhead of instrumenting the code would likely be higher than the overhead of polymorphism in a language like C#, so if their code was actually that performance sensitive, they wouldn't have it instrumented in production. If they did have it instrumented, they would likely have realized that their code wouldn't be bottlenecked by a little bit of function call indirection. Also, all the talented folk I know that stress the importance of runtime instrumentation also stress the importance of regression testing, so I suspect it would be pretty rare for a team to value one but not the other.

I'd like to take this opportunity to encourage people to learn about sampling profilers. What they do is periodically record the program counter and call stack of a process, then aggregate these to get a picture of where the program was spending its time. With a low enough sampling frequency they can have negligible impact on the speed of the software being profiled, they require no changes to production code, and they operate with fine enough granularity that they can easily tell you if (for example) a particular polymorphic function call is adding significantly to the runtime of a hot loop.

Often all you need to do to find out what the real bottlenecks are is to point one of these profilers at a running program and look through the results. Super low effort, potentially big payoff.

It's hilarious that you are skeptical of the competence of the SO core devs. You should do a little further reading, there is lots of content available online from the core devs who actually "do the work" there (ie, not the people who wrote this fluffy blog post).

I'm basing my opinion on reading the article. The claims that were made are consistent with what I've experienced working with folk with relatively limited software engineering skill. I don't think that's an unfair or mean thing to say. The whole premise of the article is to try and convince me that not following best practices can be a good thing. To me, the article seemingly reads as defensive, as if the author is actually trying to convince themselves or their coworkers that they aren't to blame for an accumulated decade of technical debt.

In reality, there's no blame to assign when tackling tech debt. Real projects are messy. Folk defer work for short term gains at the cost to their future self. Growing tech debt is just an eventual engineering problem to tackle like anything else. Best practices are distilled wisdom intended to keep those sorts of problems from growing out of control.

Re: the competency of the SO core devs, they very well could and probably are solid within their domain. Not knowing anything about them though, I think it's possible that they're a bit siloed off in their knowledge. If a lot of their core team were there from the beginning of the project and were relatively inexperienced at the time, they may not have ever invested their time into developing test tooling. In some ways it's very impressive that they're able to keep that large of a project going with so little testing infrastructure.

FYI, the first listed author of this post has the job title Principal Software Developer, which I assume makes that person is one of the people who write the code.

Edit: added missing “is”

Fair enough, thanks for the correction. I guess the tone of the post doesn't convey that they know what they are doing to some people, although it should be obvious that they didn't just sleepwalk into having one of the highest "power to weight" ratio web apps on the Internet.

I think you need both. Intuition most of the time to prevent the problem of a little bit of slow everywhere adding up, and then profiling to make sure there is no huge thing hiding in the blindspot of your intuition.

Lines like that[0] sound like the goal is to please the industry, without questioning why the industry loves these things. It's good to question established best-practices, but it's also good to understand why they are best-practices.

[0] "With our code base, you won’t be able to easily do test driven development or similar practices that the industry seems to love."

Something not mentioned but is a much bigger problem than performance for most is that if you write too many tests, you may be unable to easily upgrade/update to newer versions without a significant amount of work to update all of the tests.

You’re right to call out performance as a reason for testing. Unfortunately, though, it’s usually not that easy. A problem not found in testing may be the result of inexperience or lack of foresight in design, such as the assumption that another adapter service in front of that enterprise cloud service that only exists in production isn’t a bottleneck, or that your code that had 100% unit test coverage turned out to leave nasty half-updated data due to non-transactional services and was deadlock-prone with memory leaks.

Isn't that a bit of a red herring? Either I have automated tests that will take time to update every once in a while, or I have manual regression tests that I have to run for every code change. The automated test path seems to be the much less time intensive path, depending on how often you make big changes.

On the other hand I really believe 100% unit test coverage is a bit of a waste of time. Unit tests should be reserved for algorithmic\calculative code, more mundane code logging\carting data from a to b type code should be covered by automated black box testing.

That depends on the tests. I've seen tests that are the code is implemented like X. I've seen tests that are the code uses some API to do X. I've seen tests that are the program does X via some user interface.

What you want are tests that the program does X, regardless of what computer it runs on, or what the interfaces are. However this is impossible long term. You MUST assume something that is subject to change in the future. Some changes are more subject to change than others.

Automated tests are an assertion that "this will never again change". You can make an educated guess as to what will and will not change, if you guess right then you don't need to rework your tests. When (not if!) you guess wrong you need to rework tests.

I meant upgrading application frameworks, OS, etc., so it’s not a red herring. Those can inflict a lot of changes if you have a lot of tests affected.

Thus automated test time + automated test maintenance is usually less than manual test time + manual test maintenance. Mostly because execution time for automated tests is so much shorter than manual execution time that the higher cost of automated test maintenance is recouped and then some.

The only way for framework\OS changes to impact that is if they happen often enough to make automated test maintenance "weigh" more than all the time saved by no longer having to manually run your test suite.

I don’t think we’re speaking the same language. I’m talking about how much time it takes to change the code in every unit test, etc. because you upgraded something that requires changing the test framework, and this library and that one, so you end up not only having to fix all of your production code, but all of the tests with their mocks, stubs, injections, annotations, xml config, etc. If you have thousands of tests, this can keep people from upgrading.

> you may be unable to easily upgrade/update to newer versions without a significant amount of work to update all of the tests.

That's a code smell. Tests should only be updated to reflect a new understanding of the requirements of the system. If you're writing tests that are dependant on the implementation of code and not the behavior of code, then you're too tightly coupling your tests to your code.

>Why does the industry seem to love testing?

Because testing coverage is a typical way modern software engineers and managers show off an otherwise mediocre codebase.

>because we are better than all of you.

Well, at least they have something tangible to show for it. TDD elitism is much more ridiculous, because (as I've seen over and over) having high test coverage does not mean the code is bug-free or even properly modularized. If you want to brag about your tests, don't shot me your coverage, show me how you have 10 times less bug reports than the next guy.

The industry loves testing because the industry (big handwave, encompassing a huge pile of heterogeneous software development houses) is in the business of risk management and scaling practices.

If SO needs to double its engineering base tomorrow, can they? A well-tested infrastructure means you can trust engineers less deeply familiar with the system to make improvements without breaking pieces they aren't familiar with. The tests provide a bit of a behavior-expectations guarantee (not foolproof, as there's good language theory indicating why unit testing can't be sufficient for proof of correctness, but better than nothing). Without it, you have less safety net if junior programmers change component A failing to realize that component F deeply removed from it was assuming some behaviors of A.

One piece of this post worth noting:

> Currently, we’re trying to change this. We’re actively trying to write more tests and make our code more testable. It’s an engineering goal we aim to achieve, but the changes needed are significant.

... and that means if they have a reason to massively scale-up tomorrow, their codebase isn't starting in a good place to make that change. They've made a first-to-market risk tradeoff that they may have to pay for if they need to add a bunch of new engineers right now and have them operating independently by Friday.

> show me how you have 10 times less bug reports than the next guy.

You can't actually do that because there are too many other differences. However good tests do in the long run tend to result in high quality code.

If nothing else because sometimes the obvious fix for one bug breaks something seemingly unrelated where the obvious fix is to undo the first first. I've seen that go on for 4 rounds over 8 years in one company with different contractors doing the work every time (only by luck one of the few full time employees happened to review the code twice and think to check version control to see the history of the obvious fix).

Tests are more expensive to get an equivalent quality version one out the door in general. (if you have a 15 year development cycle maybe not). However by the time you are on version 3 you save yourself a lot of retesting the same thing again to get quality.

Though the most obvious affect of tests is as you say: a metric that management can brag about. Also one of the few metrics where gaming the metric for more isn't entirely bad (though it can be a waste of money!)

You seem to be searching out a reason to be offended by this rather bland, vanilla blog post that simply explains what they did.

Is there not a whole industry devoted to selling developers on automated testing?

100% agreed. This was a very poor article.

They've basically gone with an overly expensive Microsoft stack which meant that from the beginning they were fighting the wrong fight. Instead of thinking about scale for best performance, they had to think about best cost for minimum scale.

Furthermore they decided to bastardise their C# code to some fake low level untestable C dragon code which makes matters even worse. They have created a huge mess which is probably extremely hard to maintain and too precious to touch for a performance gain which probably still only scratches on the surface of what a true low level language could have achieved.

The best practice would have been to build the parts which matter most in a language which allows them to be blazingly fast and write nice testable and easy to understand code. Something like Rust or Go. But no, they stuck with C# for god knows what reason (maybe financial Microsoft incentives who knows?) and had to throw away every sensible coding practise in order to make it work.

This is literally a blog post describing how not to run a large scale website.

> The best practice would have been to build the parts which matter most in a language which allows them to be blazingly fast and write nice testable and easy to understand code. Something like Rust or Go. But no, they stuck with C# for god knows what reason

Whilst for work I've done approaching two decades of C#, for my own stuff I have largely used Go for years now so I'm far from an MS apologist, but that impression of C# is wrong. To be fair, the article leads partly to your conclusion so that's understandable.

C# with .Net Core, almost by design and by default, massively encourages "nice testable and easy to understand code" - much more so than most languages out there.

And unless the codebase is a hacked together monstrosity of the type that can be created using any ecosystem, .Net Core and C# are massively performant (pre-Core .Net was just about up there with some competitors but Core is in another league over the old Framework).

It's a shame that it doesn't have the mindshare outside of enterprise that it deserves, as C# on .Net Core basically gives you (practically, not syntactically) type-safe Node in terms of async/await, Go-like performance, Java-like power, and Ruby-like simplicity. Though to be honest it can take a fair while to advance from "hello world" given the atrocious naming and versioning mistakes Microsoft have made and how that impacts documentation and research.

> Go-like performance, Java-like power, and Ruby-like simplicity

You massively overestimate C#. Microsoft devdiv marketing is working.

> You massively overestimate C#. Microsoft devdiv marketing is working.

No, I don't. And it isn't based on marketing.

I've done C# since around 2001 and Go since around 2015 (in both cases tools, sites, and APIs) so my perspective on relative performance is from first-hand knowledge.

As regards Java I don't think the comparison is controversial, and I've done Ruby (and some Rails) on and off for many years and whilst some of it may be down to familiarity, I still find C# (after acclimatising to the .Net/Core Framework) to be equally readable, flexible, and simple.

You're totally free to disagree of course, I'm just taking the opportunity to correct the impression that I'm "overestimating" C# based on marketing when in reality there's no estimation happening at all - I'm speaking from personal experience.

> But no, they stuck with C# for god knows what reason

Mainly because Rust (2010) and Go (2009) weren't around when SO was first created (2008).

C# is one of the best choice at the time (and still). Adopting Windows at the time is trade off.

If that’s true why has nobody (who isn’t Microsoft) ever build anything large scale in C# before or after StackOverflow?

What gives you the idea that that's remotely true?

Reply on news.ycombinator.com