The law, despite being unconstitutional, allowed this, so you can't go back and arrest people who weren't breaking the law at the time.
What you can do is track back every arrest that resulted from one of these searches, and ask for all charges and convictions to be vacated/overturned because the evidence was collected in an unconstitutional way.
I'm more interested in damage control than revenge on this one.
These people didn't swear an oath to the constitution, the people who wrote the unconstitutional laws they are enforcing did.
You seem to be implying a different model under which unconstitutional legislative laws would remain in effect, but the people who passed them would be personally responsible for having gone against the constitution.
I don't know, it depends on their mood I guess? I'm unaware of the U.S. legal system specifically, but the Court of Justice of the European Union invalidated laws retroactively in the past. This isn't exclusive to EU laws either. My home country did the same thing with national laws. So I don't know, maybe the US can do that as well, but people just assume the US can't because it feels intuitive for it to be that way?
There are ... some exceptions. Pardons can be granted (presidential or gubernatorial) for those convicted of specific violations, or under procedures later discredited. This doesn't of itself provide the ability to prosecute bad cops / bad prosecutors, but does at least undo some of the damage.
E.g., President Biden has given broad pardons for marijuana possession under US and D.C. (administered by the Federal government) law:
<https://www.npr.org/2023/12/22/1221230390/biden-pardons-clem...>
This doesn't include state convictions for drugs possession (the vast majority of such convictions) but does signal to states which direction the Federal government is leaning.
But your damage control is within reach.