How Balanced Automates Testing and Continuously Deploys
blog.balancedpayments.com
blog.balancedpayments.com
NmQ2ZjYzMmU3Mzc0NmU2NTZkNzk2MTcwNjQ2NTYzNmU2MTZjNjE2MjQwNjU2MzZlNjU3MjY1NjY2NjY5NjQ2MTY1NmI2MTZkNmY3NDc0NmU2MTc3Njk=
My statistical background have always questioned me, are these guys, who are posting rebuses on job pages are actually searching for rebus solvers or need their job done?
Or did they ever conduct a practical coding experiment with applicants who love rebuses and who dont..
I'll give you a billion dollars for it.
when you say 20 seconds did you reach for a bash command ? interested knowing which.
so much for import base64!!
curl -s http://blog.balancedpayments.com/balanced-payments-operation... | html2text | grep -P '[^.*]=$' | head -n 1 | base64 -d | sed 's/../0x& /g' | xxd -r -c 100 | rev
I really wouldn't hire someone who wants to use grep to parse out html[1] :P
Then again, I would not like to work for a place where engineers don't understand this ;)
I wonder if the results of that study would show whether it makes any difference. :)
http://blog.factlink.com/post/45861768695/yolo-spend-less-ti...
We use git, so it's easy to revert commits, or push an update with '-f' that resets to an earlier version. Once we push this new commit (that simply puts us back to an earlier state) to our release branch, it's picked up by the testing system just like any other commit and pushed out.
(Please see my other response to a similar question, as well)
I'm guessing we could use Jenkin's join plugin for this and have a job that just waits for x minutes but time does not necessarily correlate to a feature being used.
It helps to look at what your past failures have been when going live in production to know what to audit during incremental rollout. In our case, /ping has stopped responding before, but some other common cases were increased 500 response rates or severely decreased performance across average or peak response times. These can be used as metrics when we automate. It doesn't help much for infrequently-used features, but I think the main idea is to prevent all of your instances from going down/going haywire at once together, which is unlikely to be a problem caused by such features even if you rolled out to all at once.
Would be interested to hear if you get that working with Jenkins to handle incremental rollout, I haven't tried yet. I'm not even sure how this would be done in Fabric but that might be another option (and have Jenkins call Fabric and block on its completion).
Can you reach out via support @ balancedpayments.com? Come hang out on our IRC - irc.freenode.net #balanced.
Would love to talk to you more!
The 10 minutes is to run the full suite of tests. In the case of an emergency, if we needed to get code out and wanted to bypass tests (e.g. reverting to an earlier version), we could run our deploy task manually, in parallel on all machines, and be done in about 30 seconds. This is obviously a nuclear option that we'd hope to never have to use, but if we were reverting to known-good code after a botched deploy, it might be our best bet.
This process is a safety net. If we want to work without a net, of course we have more options -- this is the tradeoff that we've made.
EDIT: small clarification
We write tests when new code is added, or a bug is fixed and we want to make sure it doesn't reappear. Hopefully we add new tests without prompting, because that's the right thing to do, but our coverage enforcement will goad us into writing tests if we forget. Ideally, new code should be testable by only a few tests -- if it requires more, then it's probably too complicated and should be refactored.
It might be a factor of testing methodology and style, many people consider that each assert is a test/each test should only assert a single things, so the number of tests grows large. I was also surprised at a "mere" 950 unit tests (our web frontend has maybe 10% test coverage and reports almost 700 tests, but that's because the test runner counts each assertion as a test, not each test case — of which we have 200)