I've lived in Ottawa and it's big selling point for me is the various national museums. I used to love going to the National Art Gallery simply because of its architecture, but I did see an excellent retrospective on Escher (including a presentation by a guy who tried to take picture of places where Escher did drawings of) and one on Renoir's portraits. Skating on the canal is pretty big in the winter in Ottawa. A friend of mine commutes to work by skating on it. You're right in that it's not as busy as London or DC, though. I consider that a plus, but that's just me.
Montreal is beautiful, but you'd better learn to at least attempt French as you'll be treated better by the shops and restaurants for it. Other pluses are: the jazz festival, the comedy festival, and the F1 race. My sister and her family moved there in January as their jobs in Toronto were being phased out.