This was my preferred approach for several years. The tests were in a repo separate from the code under test, and each declared its prerequisites, either as a feature or a version pattern.

If a test started failing, I could annotate it as an xfail with an open-ended version pattern. If I noticed that an xfail started passing, making it a upass (I think these were blue in our tests results), I put an upper bound on the xfail version pattern.

The nice thing about this was that I could always run the latest and greatest suite against any version, although it made it a little more complicated to run them as they existed when they were first run.