It's the loss of things like financial services companies based in London being able to readily "passport" their services into other EU countries without relying on local branch offices somewhere in the EU that will really hurt.
When it comes to passporting rights, it's very much in the interests of the EU to tell the UK to bugger off, and let the London HQs relocate a whole bunch of jobs to Frankfurt or Paris or Tallinn if they want to carry on doing business with EU nationals. This is likely bad if you work in some London-based back office role for a big European bank; really bad if you're a London-based fintech startup aiming to serve most European markets.