Yes, I have seen that argument often. I agree number of CVEs is a very rough metric. However, I am still confused as to why none of Chrome's CVEs would be Brave's CVEs?
The argument is, fundamentally, that it's not even that. It's not a rough proxy for the thing you want to evaluate, it's not any kind of proxy for it at all - that's a qualitatively different argument from your restatement of it.
If you want a very rough comparative proxy, an obvious one is 'Brave is a much smaller downstream consumer of Chrome, Chrome has a larger security team/infrastructure than Brave has employees'. I think you can draw more meaningful conclusions from that alone than from CVE tallies.
If I fork Chromium into my own Tomte-Browse, nobody will tag their CVEs with that.