Minor nitpick, but the L2 point is selected so it doesn't need fuel to stay there. It's like a gravity well of the earth/sun system, any object getting a little further from it will get back to it.
The unique thing about L2 is that it's the only Lagrange point where the Sun is always blocked by the Earth. That's what they're going for here.
The real reason why L2 is so interesting is that it violates Kepler's third law: despite being farther from the Sun than Earth, the orbital period is still the same as Earth, because of the gravitational attraction of Earth itself. This allows the spacecraft to keep the same distance from Earth during the year, which eases communications with ground stations.
(Disclaimer: I have been part of the Core Team of the ESA Planck mission, which flew around L2 like JWST will do.)
The real reason for L2 is that the sun, moon, and Earth are all the same direction from the telescope when it is at L2, so the sunshield can block them all simultaneously. Yes, the spacecraft will receive insolation at L2 (yay solar power), but it will always be from the same direction and therefore the instrument can be persistently shielded.
EDIT: from the the horse's mouth:
> To have the sunshield be effective protection (it gives the telescope the equivalent of SPF one million sunscreen) against the light and heat of the Sun/Earth/Moon, these bodies all have to be located in the same direction.
> This is why the telescope will be out at the second Lagrange point.
Ref: https://www.space.com/30302-lagrange-points.html "L1, L2 and L3 are all unstable points with precarious equilibrium."
Targeting a rocket with a larger fairing diameter (even at same payload mass) would make the design much simpler. An upper stage with 20 m length and 9 m diameter on a Falcon Super Heavy would allow to have a fixed mirror and heat shield that is only folded once, instead of the crazy origami that JWST is. But of course such an upper stage doesn't exist yet. SpaceX was only started 6 years after the work on JWST started...