But why not test? If we don't expect compatibility to break, then it will be little extra effort surely?
If you put it into a tier of support that states "I won't be proactively maintaining compatibility and fixing issues here", then your answer is "nothing." A failing test won't spur you to action, because of the support level.
So if the test result doesn't change your behavior, why run the test in the first place?
The Rust test job is to compile every single public open-source package on the crates.io registry. This is an amazing way to test. But it's also expensive, because that actually takes a lot of machine time.