Hacker News Reader

Comment by Arie | Hacker News Reader

Every little piece is easy/easier to test, but testing end-to-end over multiple micro-apps is hard in my experience.

Why is this? Is it simply a deployment issue - that it's hard to get all the pieces running together in a test environment?

Sorry, I meant automated testing. Getting all the piecies running in a test stage isn't hard.

In my experience even getting all the pieces installed in a testing environment is hard. Specially when you need to make sure that they are all wired correctly, with the correct version, with all their dependencies (maybe different kind of databases, queues, caching layers) and on top of that if services are written in different languages, that adds another level of complexity. Eventually when the system grows big enough and the team of people that needs a stable test environment grows big enough, it just becomes too expensive.

this is where docker and the ecosystem around it starts filling in the pieces.

also, it makes failures hurt less, and rollbacks much easier.

Docker answers half of the issues: deployment.

The other halves: distributed logging (how do you trace a 'transaction' that is being executed against multimachines/multiplaths?)

What about rolling releases? what about versioning?

What about service discovery (still immature field, look at how many products/tools out there trying to be THE service discovery choice).

We actually use syslog-ng + Loggly for distributed Loggly. We partially solve the transaction problem by slapping a request ID on every action that's initiated by a web request. We then carry this request ID from service to service, which gives us the ability to trace exceptions cross-service. It's obviously not the perfect solution, but it has helped us many times when debugging customer-facing 500 exceptions.

We solved the service discovery problem by simply using DNS. Every service runs behind an (internal) Elastic Load Balancer, and we let the load balancer configuration figure out which instance is up and which one is down. Again, not the perfect solution, but it works great for now and is very easy to setup/maintain.

Reply on news.ycombinator.com