I don't think anyone is arguing that the government shouldn't be responsible for certain infrastructure, it's the degree to which the government should be involved, and what its role should be. If I put on my more conservative hat for a minute to make an argument, I think the idea is that the government should be somewhat of a last resort for things, and that less is more when it comes to government involvement.
Part of the reason for that is that once the government is involved, it changes the incentive structure quite a bit. Using other infrastructure projects (e.g. roads, bridges, etc) for examples that others have pointed out, they're a great way to raise taxes earmarked for those projects, but then that's when politicians start to play games and move money around to their own projects, and now the roads are falling apart, so we need to raise taxes to pay for the roads!
It's also somewhat tricky because once something is a government project, it's basically that forever. It's fairly rare for something to go back to the private sector once it has become a government program. What is more likely is that the solution is more government involvement, and that's something that conservatives tend to view as a bad thing.
Obviously I'm not speaking for all conservatives, and this isn't even necessarily my own viewpoint, but I wanted to push back against the idea that "conservatives just want what's worst".