This is a fools errand. A simulation sets the ground rules of all reality including math. If the simulation was set up to make your math prove it is not in a simulation then it would simply be so. It could even be naive to believe math as we know it exists outside of a virtual world. Even if a human, camera or computer was observing an obvious glitch in the system that data would just be ignored. If the error in the virtual world were so severe that the ignore logic was failing then the individual when observed by others would be committed to an asylum and force fed drugs. If someone managed to prove to others that this were a simulation it may simply reset and be replaced with a more complex reality. A reset may have already occurred any number of times.