How We Built the Software that Processes Billions in Payments
braintreepayments.com
braintreepayments.com
More specifically, it gets across:
- the company culture
- the development practices/methodologies in place
- what's exciting about the work/company
- what technologies they like (and the fact they use a variety of technologies)
- the fact that they get that typical job adverts suck
... all while looking like a regular article.
Great work.
It seems to me that the practices you have--TDD with full code coverage and regression tests, pair programming everything, many client libraries--would lead to a massive amount of code that needs to be maintained.
In my experience I've found that TDD often leads to a huge amount of code that needs to be rewritten if you do a major change to your codebase. If you have a new pattern you want to implement it may require rewriting dozens or more tests. Also, when I'm working alone I constantly go back and clean up functions and what not to make them shorter and clearer. I've found I do that less while pairing because you want to keep moving forward.
My opinion is probably just biased by the fact that I've done relatively little pair programming and TDD. I'm sure the more you do the better you get.
But I'm curious if you could take some about the size of your code base and how these practices effect that?
Anyway, a week later he get's a call that some fairly basic functionality was broken in production which he fixed. Afterward he asked some people in QA how it ended up in production so quickly and how they had missed such a basic issue and their response was "Ops, we stooped testing your code a few years ago this is the first time it bit us."
TDD always struck me as an attempt to answer to the question what do I do if I don't know if this actually works and nobody is going to QA this crap. But, if you actually consider what happened TDD was unlikely to help because the coder was simply unaware that existing code was broken and needed to do something else.
The "cost" of high-coverage automated tests is overblown, IMO. The few extra minutes spent writing test code is worth the extra safety and confidence. However I believe striving for 100% coverage on every method/branch is overkill.
It doesn't have to be either you do TDD religiously or you have no tests.
I do think it's more than a few minutes though. The problem is when you do a big rewrite you've now sometimes got a lot of failing tests. Do you go back and delete these tests and write new ones? Or refactor the test code? It's just a big distraction because sometimes you're "holding the program" in your head, as PG would say, and to worry about tests might cause you to lose your focus on the more important code.
Bravo. In my experience that can be overkill, but with finance, I agree: why risk it. TDD everything.
We don't have a QA team.
WTF?
That might be terrifying when you consider the type of software that we're building, but we're confident that our automated testing is thorough and will catch any regression bugs.
Are regression bugs the only kind of bugs?
We use continuous integration to test every version of every client library against our gateway.
What happens when someone uses your client library in a way you didn't anticipate?
What happens when johnny-botnet hits your API directly without using the client library?
I spent several years developing games, with QA teams that outnumbered the developers. The QA team did not just play the levels through and say "it works!". Sure, they did that for the first hour. Then they'd start doing all those things that they thought someone might try (e.g. in a fit of boredom, or for a laugh). After that they'd just try breaking stuff. What a lot of bugs they would find!
As I write I find it hard to believe that I, a game developer, is having to explain the importance of QA to a financial company.
What about all the cases you forgot about. By definition you forgot about the tests too.
What about all the interactions between module A & module B? They may work 100% on their own, but not together.
A good QA department can generate many more tests that the developer could, and do so without the 'author blindness' that is somewhat unavoidable when writing code.
Here's how I'd rate the "dangerousness" of a bug:
1. How easy is it to detect?
2. How easy is it to reproduce?
3. What are the consequences of it occuring?
4. How likely is it to happen?
So all the other bugs that your system has right now, are the ones that are left: hard to detect, hard to reproduce, and don't always happen. Now turn on a thousand users. How many users are you hoping to have btw?
Your worst kind of bug:
* Is not detected for months.
* Unable to reproduce.
* Company killer. (Reputation, lawsuits, whatever).
* Happens once every 40,000,000 sessions.
I'm not suggesting that QA is useless, I think QA should guide developers in terms of testing, as in QA should help writing the test-cases including the corner-cases in spec and let the developers write more tests around those things.
I also think that QA should help performing benchmark tests, load tests, and probably write end-to-end automation-tests (what do they call it these days? Acceptance tests?)
Last but not least, QA should redefine the software processes if bugs happened regularly in a particular area. Consider QA to be a manager that responsible for the productivity of your software team: if a software process doesn't work (let's say one day you found out that TDD doesn't work well), QA should detect that and figure out a better way.
Unfortunately, QA these days are still old school button clicker and test-case fanatics (i.e.: prepare 1000 test cases and ask the director for a week to run them all).
But at the end of the day, bugs exist. No amount of human or practices would cover those exotic bugs.
I'd love to hear some more about pair programming, has anyone here done it extensively enough to shed some light on the pros and cons? My gut is that it would be less productive than a simple code review procedure, but does it reduce bugginess to a level that offsets that productivity loss?
For payment processing, that's a good thing.
As I understand it, the productivity isn't too bad, as the programmers egg each-other on. Sort of like having an obnoxious micro-manager over your shoulder, without the obnoxious bit.
In these situations the actual typing occurs after the programming has been done.
I find it increases productivity because:
* Every problem that comes up is solved more quickly with two minds on it discussing the problem. (Often more than twice as quickly than working alone)
* One person is usually the driver (uses the keyboard) and the other the navigator. This means one person can keep in mind the overall picture of what they're building, all other parts of the codebase that need to be updated and so on, while the other person concentrates on the nitty gritty of implementing a specific method.
* You get distracted less easily (IRC, Twitter, HN, etc) when you're both working on something.
* You produce better quality code - There's no temptation to cut corners when the other person is right there when you're writing it - Your pair can help you stick to TDD properly - Both people have insights into best way of writing things - This means less time wasted refactoring later
This last point is why it is so much better than a code review process. Once the code is written there is a temptation to not have to go back and change it, especially for a major redesign that someone else spots. Changing things involves risk of introducing new bugs. There's also the risk of offending the person who wrote the code
For the other 80% of what goes into any software: total waste of time, or worse. Two people, independently, could get twice as much done. But worse: pairing on something like that can result in over engineering, language battles, just about any kind of drama to make the day more interesting. Seen it happen.
If I had to choose between 100% pair-programming or 100% single-programming, I'd have to choose pair-programming. In this artificial comparison, pair-programming is more productive. Which is why, I think, so many companies fall into this trap. 100% PP teams beat 100% lone-coder teams.
However, teams that pair when appropriate will destroy the zealots of either denomination. Rationality vs religion.
>we pair program to write all of our software. We work on Mac Pros with two keyboards and two monitors. We work in an open team room; no cubicles or private offices.
No thanks, if I'm the developer. And if i wouldn't do it, why would i make my employees.
When you're pair programming, you're not wearing headphones and "getting into the zone". You're openly collaborating, sharing thoughts, bouncing ideas, prototyping things on a whiteboard, and so on. Just like at a party, you subconsciously filter out the surrounding noise when you're talking to your pairing partner. An open office is perfectly fine for this. I didn't think it would be a good situation at first, either. :)
However, I personally do not enjoy pair programming. There's a few reasons why, but the big one is that it's mentally exhausting. Eight hours of engaging in conversation completely wipes me out even if it results in amazing code. I couldn't deal with it any more. To a lesser degree, I do not get the same sense of accomplishment from completing tasks when pair programming that I do from completing tasks by myself.
That said, if I ever run a company with a handful of programmers or more, I'm going to hire engineers who like pair programming.
I don't mean to complain - when you're so focused on other parts of your business, it's easy to let things like preparing your front-end for heavy bursts slip by.
It's a bad idea for developers to QA their own code for a number of reasons. 1) Developers have cognitive blinders, like everyone else. They might not test for something that they is unlikely 2) Some errors can be impractical to catch outside of top level integration testing (getting into unexpected states in state machines or race conditions) 3) There is a conflict of interest between deadlines and meeting requirements.
I have seen companies reach 100+ developers using this approach, conclude it's unsustainable, and be forced to make exceptional efforts to build a QA team. I believe reason 3 is the biggest risk.
On the other hand, they're clearly doing something right, so far.
I wish them continued success, but I encourage them to start looking for a rockstar QA/release engineer.
Why is it necessary to suspend traffic to make these kinds of changes? Just curious.
I don't understand why you'd need a custom HTTP server. It might be overly simplistic, but just write a module for nginx? Surely that would have been faster. You get controllable timeouts, you could tweak their throttling code to stop it throwing 50x, and modify/take inspiration from nginx-ey-balancer. And you get a bullet proof HTTP engine for free?
That's one thing you can't seem to do.