If you don't, you get the exact same kind of question with ARM as with RISC-V. Do you use NEON or with SVE? Or do you conservatively compile without vector instructions at all even though it could possibly result in speed-ups for some loops?
For example, can you use FEAT_CSSC to improve code size and performance? Well, if the target is <v8.7, the answer is no. If it's v8.7 or v8.8, well, then it depends on whether your specific implementation has it. Only after v8.9 is it mandatory.
Targeting armv8a is the moral equivalent of targeting RV64GC insofar as it will run on any application class core. Targeting that, however, leaves a fair bit of useful ISA enhancements on the table, and so you tend not to want to do that if you can get away with it.