>> So the first request takes like 5s or something.
I haven't worked with IIS in more than five years, but couldn't you change some setting to infinity so the thread never sleeps... or something like that? I remember the "5 second" thing being a problem with commercial IIS apps we deployed, and that's always how we avoided it.