Hacker News Reader

Comment by jtms | Hacker News Reader

I have not used it, but I have heard this is a very useful tool https://github.com/Netflix/chaosmonkey

Sure, often at a super initial stage the people running the test can just do this manually though. Don't be put off by having to set up the whole suite, a lot of the value from Chaos Engineering can be achieved from randomly removing bits of infrastructure manually (a d6 and a lookup table works fine). The value comes from the learnings as a result of infrastructure being terminated.

What would be advised in the following situation if one wanted to follow chaos engineering principles:

- there's a service that needs config data from a DB on another node to initialize itself to become useful - should the service die if it doesn't have connection to the DB on startup (so that the error propagates), or should it start and perform retry indefinitely until DB connection is set? Until that happens it sends back error code to its consumers.

I don't think what to do during the failure case is part of chaos engineering.

identifying that it needs to do something, and whether it does it or not is part of chaos engineering. eg by turning off the DB for a bit and seeing what it does

Basically it randomly kills prod nodes and forces engineers to consider this when designing their services. This is one of those "so stupid it's genius" ideas, kinda like mailing DVDs rather than having people to go to a movie store.

Reply on news.ycombinator.com