The biggest downsides are speed and climbing ability. 80k or so has been a reasonable max distance on tour (I've done one 100k day, it was long) and I wouldn't take it to the Alps.
Like the OP, I run Schwalbe Marathon Plus which has been good. But I have had one catastrophic puncture after riding over a particularly nasty piece of glass that cut straight through the tire. After that I bring a folding backup tire.