Sounds like you're testing wrong then ;)
Mock the entire socket API, so you're not actually testing your software?
At the end of the day you need to run some tests with external interactions, and you shouldn't cache those.
Also keep in mind any time you perform syscalls, you have external interactions. Their behavior could change and you want to know if they have intermittent failure scenarios. Using threads? You're nondeterministic, you should keep running tests.