SDL should probably use inotify() on linux so the kernel can let it know when /dev/input has changed rather than polling it.
It seems like Papers, Please uses a statically linked version of SDL, without udev or inotify support compiled in.
1: https://github.com/libsdl-org/SDL/blob/d0de4c625ad26ef540166...
Edit: though here if it's a problem of file enumeration and access, I'd probably just LD_PRELOAD something to bypass libc file access functions and return the same result than the first time, with no delay.
SDL does have a built-in way to do that trick. A quick web search tells me it's called SDL_DYNAMIC_API.