msherry

When I saw this, I first thought it was an extension of https://news.ycombinator.com/item?id=5326159 ;)

More like SiftScience is a weaponized version of something a lot of companies develop in-house, I would say.

I don't want to hijack this thread too much from the original post, but we use some of the same software as Airbnb (scikit-learn, randomforest models, etc.) as well as some stuff developed in-house. Credit card fraud has been one of our biggest issues, and we've developed some pretty robust systems to fight it. Contact me privately and I'd be happy to talk about it -- this stuff is what I do for a living.

(I don't work for Airbnb, but work in a similar space)

Yes, this variable is usually set after the fact. For instance, a given transaction may have led to a chargeback, or may be done by a known fraudster. These models are usually trained on historical data, so we can know with some certainty which transactions are fraud.

It could be that a transaction was fraudulent but has not yet led to a chargeback (maybe the real cardholder hasn't yet seen their statement?), so there's still some uncertainty, but hopefully that approaches a minimum after some time passes.

I think you raise a good point here -- Gittip is run by committee and consensus, via Github issues and pull requests. The Linux kernel is developed via community as well, but has a strong leader at the helm to direct things. In contrast to this, Gittip feels like a bunch of people kind of doing their own thing, which leads to a general impression of aimlessness.

I suspect that branding and marketing efforts shouldn't be run by +1's on issues, but by someone taking charge, picking a direction, and running with it.

That's always been my favorite response to him, as well. It made me tear up a bit reading it.

Here's the thread, for anyone interested: http://www.metafilter.com/119424/An-Operating-System-for-Son...

Did you read the article?

"From a data-driven perspective, this is nothing short of maddening. There are many ways to board a plane, with “back-to-front”—the chosen boarding process of most U.S. carriers—the slowest."

They're worthless because clearly no one is using them. They may theoretically be useful, but it doesn't matter if they're never put into practice.

Sure, but I if as an inventor can't be guaranteed some protections for my work, for at least some period of time, I'm less likely to produce this better form factor in the first place, which is a net negative for consumers.

Yup -- you can authorize a card with a missing, or even incorrect, CVV.

I work at Balanced payments, and this frequently trips up a lot of people. If incorrect CVV information is entered, we note that in our response, but it's up to the user of our API to decide if they want to accept that card or not. (We also use this information as one component of many in our internal fraud systems, of course).

I get that you were being flippant, but apparently Turkish is unusually hard to localize for: http://www.codinghorror.com/blog/2008/03/whats-wrong-with-tu...

http://qntm.org/responsibility is an interesting go at it, and where I first learned of the concept. His other fiction is pretty fun, as well.

Same thing happened to me. Presumably I can restore the older version from a recent backup, but this is pretty weak.

Ahh, I see. Right, 950+ is the number of unit tests we have, each of which contains a number of assertions. If we count those, it's well into the thousands, so I guess the original question wasn't that far off the mark.

Hi there. Why would you expect 5000+ tests, or anything more than 100? It seems like the number of tests that is the "right" number would be very dependent on the size of the codebase, which wasn't mentioned in the article.

We write tests when new code is added, or a bug is fixed and we want to make sure it doesn't reappear. Hopefully we add new tests without prompting, because that's the right thing to do, but our coverage enforcement will goad us into writing tests if we forget. Ideally, new code should be testable by only a few tests -- if it requires more, then it's probably too complicated and should be refactored.

If we're already at the "Oh shit" stage, then we might be happy sacrificing test coverage to go back to known-good code, in which case we would run our fabric `deploy` task and have our rollback completed in ~30 seconds. The idea behind this process is that we never have "oh shit" moments like that :)

(Please see my other response to a similar question, as well)

The whole idea behind this testing infrastructure is so that we're fairly certain that any deploy isn't going to leave us scrambling to revert to an earlier version. Since we've implemented it, we haven't had to do that yet -- it's caught all manner of issues large and small that we're glad never made it into production.

The 10 minutes is to run the full suite of tests. In the case of an emergency, if we needed to get code out and wanted to bypass tests (e.g. reverting to an earlier version), we could run our deploy task manually, in parallel on all machines, and be done in about 30 seconds. This is obviously a nuclear option that we'd hope to never have to use, but if we were reverting to known-good code after a botched deploy, it might be our best bet.

This process is a safety net. If we want to work without a net, of course we have more options -- this is the tradeoff that we've made.

EDIT: small clarification

Hi there. I'm the author of this post.

We use git, so it's easy to revert commits, or push an update with '-f' that resets to an earlier version. Once we push this new commit (that simply puts us back to an earlier state) to our release branch, it's picked up by the testing system just like any other commit and pushed out.

This is basically what we did, except during the "upgrade the application and database" step, we suspended all traffic to our app servers. The schema change was an incompatible change (I think this is what you mean by "resolve data continuity"?) So basically, our old code and new code could not run simultaneously, because they were designed against incompatible schemas.

Thanks! We had been practicing on test servers and so we had the commands ready to be repeated, but you're right -- for reproducibility, a script would have been the way to go.

You're definitely correct that bugs in the new application code would be a showstopper. Another bit of infrastructure we plan to write about if there's interest is our testing setup.

Basically, before any deploy (not just drastic ones like this), not only do all existing unit tests for the new code have to pass (with a certain minimum code coverage threshold), but also a full acceptance suite, which tests the new code and how it connects to all of our other bits of infrastructure. We simulate an extensive set of operations that a client might perform against a test instance of the server, and also run a number of tests with all of our services loaded in-memory, which allows us to mock/patch arbitrary points in the code, to assert that what we expect to happen is actually happening. We also run each of our various clients' test suites against the new code, to make sure that each client sees the behavior it expects to see. This testing suite has dramatically increased our confidence any time we have to do a deploy, and best of all, it's all done automatically.

You make a good point re: fault tolerance. As it happens, I've simplified the diagram quite a bit to make it simpler to visualise. We have more than 2 Nginx instances, and more than one shard of our app was running the new code.

We actually have much the same functionality. We can dynamically route/rate-limit requests on a per-marketplace (or anything else, really) basis, using Nginx's Lua integration capabilities. We may end up writing another post on this, if there's any interest from the community.

Sure, this definitely isn't the right approach for all migrations everywhere. When we have migrations that we know will take a long time, we try to make them non-invasive enough that we can have code that works with old and new schemas simultaneously. That way, the migration can take as long as it needs to, and everything Just Works™.

Are you saying it would work without code changes by increasing the timeout limit? We had to deploy new code to work with the new schemas, since the underlying models went through drastic changes. Having our code be compatible with old and new schemas simultaneously, as it is during most of our migrations, would have been extremely difficult in this specific case, which is what prompted this solution. I don't think this is a problem that could have been solved by simple timeout changes, but I'd love to hear your thoughts on it.

Amazon being PCI-compliant was a requirement for us using them in the first place :) We could have possibly made a case for their PCI-compliance obviating the need for us to do our own SSL termination, but that could have gone either way, depending on our PCI audits.

Using Nginx also lets us do fun stuff with routing using Nginx's Lua integration, which we may end up writing about in the future as well.

Due to the fact that we process credit card payments and thus fall under PCI scope, we have to adhere to the PCI DSS (data security standard). There's a "quick" summary of it here https://www.pcisecuritystandards.org/documents/pci_ssc_quick... , and section 4.1 in particular specifies that we have to secure cardholder data all they way to our servers -- Amazon's ELB doesn't quite count.

Nginx is primary for SSL termination and static assets. At the time we set up our infrastructure, I don't believe HAProxy supported SSL termination. According to Willy Tarreau's comment to the first answer of this question (http://serverfault.com/questions/426919/should-i-use-an-ssl-...) it was added in the same release as the patch I mentioned, coincidentally.

Absolutely.

Being a payments-processing company, we have a variety of users (marketplaces), each of whom has users who are widely geographically distributed, so our usage doesn't drop off as much as some other types of sites might on weekends. That being said, on weekends we see slightly lower traffic than during the weekday, so we performed this migration on a Saturday evening.

Then some of our clients would have been disappointed by the timeout errors they had started seeing, if they happened to make a request at the very start of the migration ;)

We ran our migrations multiple times on test instances of our database, because we were worried about this exact issue. We optimized the migration to remove extraneous changes a few times in order to cut down the time taken. Also, 13 seconds was actually the upper bound of what we saw -- many times we ran it, it took closer to 9-10 seconds.