I recently upgraded to my "dream" daily driver car that came with 22" rims from the factory. It's definitely the worst part of the vehicle. Unfortunately I can't go much smaller, as 21" are the smallest that fit over the giant brake calipers.
Due to the weight of the vehicle, the larger brakes are likely needed - but the difference between this "performance" version of my car vs. the "luxury" version built on exactly the same body/chassis but with 19" wheels is night and day for day to day ride quality on poorly maintained city streets.
Once you get on fresh blacktop on the highway and start taking corners at a high rate of speed it's a different story, then the car just glides and "sticks" to the road super well due to the giant (wide) wheels. But that accounts for perhaps 5% of my driving if that.
I know a couple people who have done it. You get much better ride because you have taller sidewalls, and you're much less likely to destroy a tire when you hit an object. These massive wheels on cars now are totally ridiculous.