The payload on the missile doesn't need to be a nuclear weapon of any sort. The important thing about the ICBM is that being further away doesn't stop it. Nuclear weapons are the obvious choice because it's hard to imagine why you want to strike something so very far away, at such great expense, with conventional explosives.

The German V2 rocket from World War II has a maximum range of about 320km. So you literally can't fire one from say Berlin to London. They were actually launched from coastal sites in the Netherlands and other occupied countries, and as the Allies took territory after Overlord, the targets changed to cities nearer Germany because the launchers were pulled back.