If researchers would stop voluntarily sending their papers to private, for-profit journals then this wouldn't be an issue.
That way you can still use historical publications to evaluate researchers on day one, and the evaluations will be accurate assuming most researchers historically each published the same percentage of articles in open access journals, but nobody will want to publish in anything but an open access journal going forward.
Journal reputation is a rather effective filter on paper quality, and the open access journal business model has a very uphill battle to fight, since it looks very close to the scumbag, predatory "you pay, we publish, no questions asked" [1] model of garbage.
[1] Really scumbag: "You just literally copy-pasted someone else's entire research paper and slapped your name on it? We don't care."
This argument is quite common. I actually don't agree with it. Right now there are many second-tier and third-tier outlets that lock up research that nobody reads. Nobody would be denied tenure or refused promotion for not publishing in those journals. By publishing your lesser research in open access journals, there's a chance someone might read it. You could build an open-access infrastructure there.