You're saying we shouldn't spend time to make sure the algorithm is mathematically sound...because the implementation will have side channels? Using your metaphor, that's like saying we should never bother designing a secure lock because someone will just break a window to get in anyway.