I had also heard pre print suggestion that those who had only acquired omicron without prior vaccination or delta may not have good neutralization of delta.
https://www.medrxiv.org/content/10.1101/2022.01.13.22269243v...
This wouldn't seem to bear on what you have said because you were very specific about antigen testing and a doctors letter that you had recovered from covid but people are fantastically bad at making such fine distinctions about things they poorly understand and many who are unvaccinated seemed sure they had had covid even when the illness they proposed was prior to the advent of covid in country. Their logic is in general so bad that they literally changed the date of covid impacting their nation to account for this disparity instead of concluding that their illness back in 2019 was unlikely to be covid.
By January 2022 everyone who hasn't vaccinated has "had covid" whether they have actually had it or not and any differential in public policy towards unvaccinated people who have recovered is likely to result in disadvantage for society at large.