Plenty of unskilled workers are from other parts of the European Union. I doubt refugees make up significant numbers of skilled workers in the EU.
I think this mainly applies to the U.K., but I'm mainly basing that on anecdotes and the last time I looked a the numbers (ages ago) so I'd be happy to be corrected