This is what I saw from my roof a couple days ago: https://twitter.com/aaronmhamilton/status/919820166833242112
The only sorta crummy building on the waterfront from this vantage is the Canada Malting Silos, but even that has its charm.
To me, a lot of the rest of the city is an eyesore. Unfortunately a lot of the large buildings in Toronto are brutal postmodern slabs of concrete spiritual oppression, including many around New City Hall.
The Port Lands are not actually so bad at the shore, some of the industrial stuff like the Hearn coal reactor and the recycling plant that went up in flames a month or so ago is a bit rough, but it's convenient to the city that these things are within city limits. It is exceptionally unusual to have facilities like those on the Port Lands and the island airport this close to the city center.