FWIW, I think best practice here is to hardcode all feature flags to off in the integration test suite, unless explicitly overwritten in a test. Otherwise you risk exactly these sorts of heisenbugs.
At a BigCo that’s probably going to require coordinating with an internal tools team, but worth getting it on their backlog. All tests should be as deterministic as possible, and this goes double for integration tests that can flake for reasons outside of the code.
That way when you get a failure, you can reproduce it. And then one of the easy things to do is test which features may have contributed to it.
Nothing kills confidence in testing more than test flakes. It’s a huge drain on velocity and morale, and encourages devs not to trust test output.
If you want to have some sort of chaos monkey process that runs your test suite flipping feature flags at random and notifying teams of failures (along with some sort of resourcing to investigate) I could get behind that. But that should be something outside of the main suite that gates code deployment.
If a test passes when run by a dev pre-commit, it should pass in CI.
Yes it's kinda fractal of bad practices that have to align for this problem to occur, but that's the nature of tech debt.
He’s not saying to alter any of the feature flags used for the test, but simply to record which were used during the test.
Simply logging doesn’t introduce any of the issues you are describing.
> I think best practice here is to hardcode all feature flags to off in the integration test suite
That's pretty clearly about forcing the flags to be off, i.e. altering them, and not about logging their values.
If you’re testing a new feature, you should have explicit tests for the enabled state (along with existing tests for the disabled state).
If you have bugs propagating up the stack from flags changing in low-level dependencies, the change to the dependency is probably not properly tested.
Alternatively, if the feature flag gates a change to the interface of the dependency, you should have explicit integration tests covering the systems on both sides of the change.
One way I have seen this handled is to enforce restricting rollouts of a feature flag to 95% at most. That way turning a feature all the way on requires removing the flag from your codebase. It’s draconian, but honestly anything less than that leads to the situation you describe.