> Otherwise why do finance have to pay engineers more (at least here in London)

Because of the demand and supply curve. If you require people in the city every day your hiring pool is less, if you require SC clearance your hiring pool is smaller, if you require some finance knowledge or experience of working on a trade floor then your hiring pool is smaller.

    > or why are the interesting jobs nearly always paying below market?

That’s not true, unless you refer to startups by “interesting” jobs and then part of the deal is equity with a chance to make the engineers retire 30 years earlier than normal workers. Not a risk worth for everyone but it’s there for those you want it.