The issue here is that polish workers are putting downwards price pressure on the construction industry. If Britain leaves, that will no longer be the case. Leading to increased prices for construction, which leads to less construction being done. Which will slow down the economy.

It is not as simple as replacing blue-collar workers with white-collar and expecting things to turn out awesome. The demand for blue-collar workers will remain. People in Britian will in the future get less for their money (this will also make it a less attractive destination for foreign educated professionals)