From personal (professional) experience, our current visa and immigration system doesn't always work very well. It is slow, expensive and overcomplicated.
Inevitably it also sometimes winds up with a wrong decision due to some technicality or administrative error, and I've heard nightmarish stories from friends who had to deal with that situation because of some issue at work. Some received advice about not using the official appeals process, because it's cheaper and faster to just start over. I've even heard of company reps having to go abroad themselves to resolve some visa problem with a foreign worker because there was no process to speak with anyone within our own government in our own country who could sort it out.
This is not to say that the general principles the system is supposed to operate under in theory are necessarily bad, or that no-one can successfully navigate the system in its current form; obviously hundreds of thousands of people do every year. But it's not working well in practice, and right now at least colleagues from other EU member states don't have to deal with it. So there is legitimate concern that if we lose the freedom of movement aspect of EU membership post-Brexit and the visa and immigration system isn't dramatically improved, that whole group of people will be greatly inconvenienced in practice.