There are some big downsides to reducing free movement of labour. I don't think we fully understand how much the current situation has resulted from them yet though.
While Brexit is surely a major factor here, obviously we also have the pandemic that has a particularly severe effect on some of the same industries. There are other significant factors in specific industries, such as the terrible conditions for lorry drivers in the UK compared to elsewhere in Europe. There's also the bizarre own goal of the government pushing ahead with its IR35 agenda at a time like this, which again hits some of the same industries particularly hard.
It's going to be difficult to discuss these industries rationally and consider potential changes that could be made without some reasonably objective way to evaluate all of the relevant factors and how much they are contributing to the current situation in any given industry. As with almost anything to do with Brexit or the pandemic, it seems unlikely there are any simple and universally "right" answers.