I get what you are saying, but I disagree. If there were millions of invisible unicorns, surely we'd occassionally bump in to one. Azaleas would go missing. With the right technology, it might be possible to harvest them for ivory or meat or hoofs for glue.

If we live in a simulation, there may be experiments that would demonstrate the fact. There may be ways to 'hack' the simulation and break all kinds of physical laws.