Hacker News Reader

Comment by paxys | Hacker News Reader

Lots of ways to test in production. IMO the way you are suggesting – injecting synthetic data into prod – is the worst of both worlds. You aren't actually testing real world use cases, and end up polluting your prod environment.

Some common ways to go about it:

- Feature flags: every new change goes into your codebase behind a flag. You can flip the flag for a limited set of users and do a broader rollout when ready.

- Staged rollouts: have staging/canary etc. environments and roll out new deployments to them first. Observe metrics and alerts to check if something is wrong.

- Beta releases: have a group of internal/external power users test your features before they go out to the world.

View on HN

And from real-world experience:

- Feature flags should be easy to create or even automatic. Otherwise your team will bypass or forget to create them, or start reusing the same ones until the flag doesn't make sense anymore

- Don't let devs name the flags, or agree and enforce a convention. They should also make sense without context (NOT "test-fix-error")

- Remove flags over time or they will lose all meaning and context

- Categorize your flags to automatically grant them to certain roles/groups

You aren't actually testing real world use cases

You can mock them from actual data. The last system I worked with and tested in production processed 1..0.1 live requests per minute. Hot-testing it was a torture. There was a test site which could produce requests, but using it was a torture as well, also it worked only on even days of week and could only be set up on request (as in IM, phone call, not HTTP) due to administrative cf.

Before anyone asks, no, it couldn’t be tested off-production. I could spin up a “staging” server with all proper infra, but (1) after all the modifications it would just turn into production, (2) incoming data was live anyway, and was an obligation to the company (i.e. all my fails closer to a response went to support and ate limits). Peer services had no such thing as test mode.

IMO your first two suggestions aren't actually a form of testing.

If you've rolled out a feature to real paying customers (not beta testers), and find that it's broken, you don't get a pass because you were only "testing".

Reply on news.ycombinator.com