+1
At Meta we do similar heuristics on which tests to run per PR. When the system gets it wrong, which is often, its painful and leads to merged code that broke unrun tests
At Meta we do similar heuristics on which tests to run per PR. When the system gets it wrong, which is often, its painful and leads to merged code that broke unrun tests
Like anything it's about tradeoffs. Though if it were me i'd simply write a mechanism which deterministically decides which areas of code pertain to which tests and use that simple program to determine which tests run. The algorithm could be loosely as simple as things like code owners and git-blame, but relative to some big set of Code->Test that you can have Claude/etc build ahead of time. The difference being it's deterministic between PRs and can be audited by humans for anything obviously poor or w/e.
As much as LLMs are interesting to me i hate using them in places where i want consistency. CI seems terrible for them.. to me at least.
Which suggests there might be big room for improvement. If LLMs open up new ways to be more precise about which E2E to run, that is a significant benefit, because you’d manage risk but reduce the time and resources it takes.