How do you handle tests against new features that would definitely not pass on commits before the feature is introduced?
In general, I want to store one or more services and a first/last valid commit for each one as metadata for each test so that my framework knows which ones to run based on the environment under test. Haven't added this feature yet though.
If a test started failing, I could annotate it as an xfail with an open-ended version pattern. If I noticed that an xfail started passing, making it a upass (I think these were blue in our tests results), I put an upper bound on the xfail version pattern.
The nice thing about this was that I could always run the latest and greatest suite against any version, although it made it a little more complicated to run them as they existed when they were first run.