Fit does.
The problem is the air going around the mask and through the seals, bypassing the filter entirely. As for the 99/95 in N99/N95, it is the percentage of filtration for a specific size of particle in controlled conditions, which may not be how effective it is at filtering virus aerosol particles. The N99 will most likely do better, but by how much? Just because the number suggest a 5x improvement doesn't mean it will be a 5x improvement, and all that is useless anyways if it leaks like crazy because of a bad fit.
But assuming the masks really are the reason for your lack of infections, maybe these FFP3s are simply a better fit. It is very possible as people have different faces.