Removing the memsets from the networking layer of one of my projects basically halved the frametime of that system (which was a bottleneck).

I'm all for defensive programming, but when you need to be fast, you'll just have to make sure it's correct.