Hacker News Reader

Comment by anthonyb | Hacker News Reader

> Tests don't magically help you invent features/work around introduced issues in 3rd party systems.

Uh, yes they do. You want a unit or system test which covers the case where an external system is down or returns something that you can't parse. Something like:

  # code to take third party thing down 
  # eg. mock out lib and return nonsense (unit tests)
  # or add an /etc/hosts entry (system tests)
  assert "Sorry, but that feature is unavailable." in page.content

Now the entire app doesn't asplode, and you can wait until 9am to fix it. Follow up is to make sure that you're on whatever mailing list tells you when changes are coming.

The only case that this doesn't cover is when it's a) an essential part of your app, which b) you aren't paying for and c) they don't have a mailing list, in which case wtf? you need to find a better 3rd party library/service.

ps. Look up the "chaos monkey" - it's very enlightening :)

View on HN

(I work for Netflix)

It's funny that you mention the Chaos Monkey, considering that Netflix has 24/7 on call programmers for tier 1 support.

We do however also make great efforts to make sure that we are resilient as possible to failure of 3rd party services.

I suspect you also pay for your 3rd party services, which the GP's company doesn't seem to do.

I used to work support for a SMS aggregator. The core business was delivering texts to mobile networks. Mobile networks break all the time, it turns out. And can't really be replaced - there's only one T-Mobile.

Now, when they died or sent garbage, our software wouldn't crash. But our service would stop functioning. And alarms would go off, and we would have to confirm why, and call them, and ask them if they knew if it had stopped working and why.

(The follow up was indeed to always ask to be added to the mailing list that would let us know this ahead of time. These proved remarkably unhelpful. In once case, we ended up setting up having to set up a mobile number to receive SMS alerts - this was apparently the only way they would notify anyone?)

And reasonably often actual engineers would be needed to be woken up. And in several cases, change the parsing behaviour so we could handle sudden unexpected changes in the format they returned. Yes, in the middle of the night.

Our system wasn't perfect, but I'm not sure our problem was simply a lack of system or unit tests.

During the day, we largely dealt with more minor customer complaints, and ongoing maintenance, outages and other nonsense fromt he carriers. Engineers would often have to dig into these edge cases too, and there was a nominated maintainer to look at this stuff, to let the rest of the team add features and work on long term fixes. But sometimes you just need someone to wedge LargeCustomer's encoding settings because they can't figure out how to properly specify it on their end.

a, b, and c, and there is no other / better service, so it's a difficult one to solve.

also things can't wait until 9am or the selected waking hours, we have too many people/systems relying on working infrastructure, so an issue popping up at an ungodly hour is fixed there and then, even if it means calling other people.

Those are the worst support calls, 3am on some weeknight, and you can't actually fix the problem because you're not 100% certain, and you need to call a colleague and wake them up too. You feel like an asshole.

> a, b, and c, and there is no other / better service, so it's a difficult one to solve.

You don't say what the service is, but if your company is relying on it to the extent that you need to be awake at 3am, then I suspect it's well worth calling the people providing it and offering to throw money at them.

Otherwise you're essentially relying on their goodwill for business continuity...

Reply on news.ycombinator.com