The reason refugees as a concept is particularly singled out, is because they tend to be a wave by definition, which is much more difficult to handle than routine immigration.
It's why China is so afraid of destabilizing or allowing North Korea to collapse. Refugee waves have caused immense problems pouring out of Venezuela (and into other poor or semi-poor bordering nations completely unable to deal with it). Countries like Jordan, Ethiopia and Lebanon have had vast refugee problems that have overwhelmed them very substantially.
In fact, if you go down the list of countries with the most refugees, they're all countries that are among the least equipped financially to handle it: Turkey, Pakistan, Lebanon, Iran, Ethiopia, Jordan, Kenya, Uganda, Chad.
Turkey has taken in perhaps two million refugees in just the last six years. They have a median income around 1/5th to 1/7th that of Canada or the US. They're completely unable to absorb that many refugees. Lebanon has 200+ refugees per 1,000 citizens currently, the notion that that isn't a problem is absurd, it's the very definition of overwhelmed.
It's anything but ridiculous, it's a not uncommon history of refugee waves spanning thousands of years.