Hacker News Reader

>And whatever amount of work you put into your test suite and automation, sometimes you cannot test some things. Sometimes you have to wait for a real day (or God forbid a real quarter) to pass.

I honestly can't imagine any software product with test automation that would require more than a real day to pass, never mind a quarter. Not unless some person or bureaucratic process was deliberately slowing it down.

>But all this has been at least partially resolved. What we are still sorely lacking is simulation. Everyone who ever worked with a staging environment knows how limited this usually is. What I would like to work with before I retire is a tool that lets me standup a complete copy of the production environment, isolated from the real world of course, with all external services automatically mocked. Now that would help to improve the quality.

There are plenty of tools that do this. I run simulators to catch sent mails, pretend to be browsers, mock REST APIs, etc. This is the bread and butter of test automation. The actual scenario writing I find to be the easiest part.

The real problem is that this stuff is usually done really badly by cheap, poorly skilled people who will never earn as much as a dev and who almost never do this part of the job properly.

The actual solution is in my username.

Some bugs/issues appear over longer time spans. Things like issues around daylight savings time. Or memory leaks (or file size limits) that build up over a long time. Or even just accruing a lot of data in a system to the point where it starts exhibiting performance issues in certain workloads.

Ive created tests that mocked all of these scenarios before. They're all hard. They require engineering skill. They all fit into the category of things that a good dev should be able to do but "wont have time for" but a QA on 40-80% of their salary won't ever manage.

The weird thing is that this really isnt a technical problem. It's an issue with social class.

To be able to test them, you need to be able enumerate them. Part of the problem here is that you don't always know what regularly occurring external thing is going to cause the problem.

It's not a skill issue when your application depends on some external API and the provider doesn't have a sandbox system where your integration tests can spin up a realistic "fake" API environment on demand.

IME almost no APIs actually provide that level of sandbox support so that immediately means you can't run comprehensive, fully automated tests of those integrations.

Of course you can mock those endpoints instead. But then you're not really testing the integration at all and conveniently ignoring the place where most of the real risk is often found.

I've faced had this exact scenario before. It was an SMS API which a major bank forced us to use. I set up a test against a mock version of the API request/response which I constructed manually from their crappy docs.

It didn't work exactly right of course because the docs weren't exactly right. Ultimately all we could do was try to fit to the docs, manage expectations, release into production, see the errors, log as much as we could, try to cope with the fallout and update the mocks accordingly afterwards. For each problem/error we ran into, this took under a day.

A longer test/release cycle would have made this problem far worse not better.

It didn't work exactly right of course because the docs weren't exactly right.

Indeed. That approach also won't pick up changes to the remote API that break your integration.

An alternative strategy in this situation if you have access to some kind of remote sandbox - but maybe just a single stateful one that must be shared by your whole org - is to factor out your integration with that service into its own component/service and deploy controlled updates to that service less frequently and after running through a comprehensive integration test using the real sandbox.

This probably involves a much lower release frequency but it also keeps your integration as stable as possible for your internal clients and tested against the real external service. If you're talking about remote services that do things you can't trivially revert - like physically transporting some object or causing money to change hands - then the strategy you described of trying to manage expectations while testing in production is probably not an acceptable alternative anyway.

This idyllic view sounds theoretical to me. What I've seen working best is a fast CI/CD pipeline with unit tests, key journey tests, and integration tests with blue/green deployments. This in my experience is far superior to staging environments for both startup and very large scales. In addition the development environment itself should be as close to prod as is practical so most checks can be done there.

I've worked on quite a few, admittedly large, projects where we simulate nearly all external interfaces.

It is mostly the human that is most expensive to replace with a simulator, apart from simple end-to-end or stress tests.

Is anyone working on a reinforcement learning agent for proper monkey testing? :)