Turbo gasoline engines were hard to make well, I gather mostly due to predetonation. Gasoline engines historically use manifold injection and spark ignition, which limits compression ratios. However, with advances in technology, particularly in engine computers & sensors, small turbo gas engines have recently become far more competitive.
So, basically, from the vantage point of 2018 turbodiesels might seem like a questionable choice- but ten or twenty years ago, good small turbocharged gasoline engines mostly didn't exist.
I believe, as you said, diesels also have lower CO2 emissions per mile than gasoline engines. The problem is the increased SOx and NOx emissions as well as soot emissions that contribute to smog.
In theory, Diesel engines should have much lower smog than gas engines because Diesel engines require excess oxygen rather than stoichiometric ammounts like in gas engines. The problem is that diesel fuel doesn’t vaporize nearly as well as gasoline, so you end of with tiny droplets of diesel in the engine rather than pure vapor. So even though there is excess oxygen, the O2 can’t get to the inside of the droplets and the engine heat basically cooks them into soot.
There was a massive push following the Kyoto Treaty to reduce CO2 emissions. From 1996, when separate emissions standards for diesel/gasoline engines were introduced in the EU, diesels initially were allowed less than half the emissions of gasoline engines, though since 2005 it's been exactly half. Combined with this, many countries moved vehicle taxation to being based on CO2 emissions of the vehicle, providing a financial incentive for diesel powered cars.
As for the cause of the various emissions: NO_x emissions are mostly a function of combustion temperature, AFAIK, whereas PM is indeed caused by incomplete combustion.
But try accelerating from every stop in the city all the way to the redline, pedal to the floor, but never going above 50km/h - you would never go above the speed limit, but the tank would disappear in no time at all. I'm certain that if I tried I could burn the entire 50L tank of the Polo in ~250km, by just keeping it in high revs and accelerating hard(as hard as you can in a 90bhp Polo anyway) and frequently. The injectors will happily keep feeding the engine more and more fuel as you do this, while diesel common rail systems have almost a hard limit of how much fuel they can inject - no matter what you do you won't go above certain values.
maybe in a frictionless vacuum!
At high speeds the force you have to exert to counter wind resistance is less than what you need to gently accelerate when going slowly.
This gives you an efficiency advantage right off the bat.