M5: 4288 lbs, 550 bhp, 400 mm six-piston fixed-caliper compound disc brakes, custom Bavarian elastokinematic suspension honed on German race tracks for decades.
P85D: 4936 lbs, 557 bhp, 355 mm four-piston fixed-calliper regular disc brakes, standard suspension setup you find in a Ford Crown Victoria.
It's not a surprise that a heavier car with the same power and significantly worse brakes and suspension handles significantly worse.
But as you say, no-one is buying a Model S to go play on racetracks. Especially since we know it can't do a full lap of the Nürburgring Nordschliefe without overheating and reducing power output.
I've been a driver for twenty years and I have never felt like I cared about 'handling'. When car reviewers talk about it I always picture a 45 year old man-child white-knuckling the wheel at the local intersection while revving his 325i, eyes darting between soccer moms and pensioners in the cars around him and imagining himself to be Ayrton Senna. Though that is probably uncharitable :)
I have owned many different cars between a top-end BMW and a beater Hyundai. The BMW is certainly a more enjoyable ride, but 'handling' was never a concern, except perhaps in a very occasional circumstance that might have gotten me arrested.
At worst, "handling" is the difference between a car that's fun to drive and one that bores me. At best, it's the difference between smashing into the truck that stops suddenly at highway speeds and making it onto the shoulder.
I drive a small family car and I know that with worse handling we'd all have been to the hospital at least twice.
This is true, and the U.S. road network evolved this way, with cars evolving to suit the 'freeway' or the little road to get to the 'freeway'.
In Europe we have an entirely different type of road network, ours does have fast motorways and dual carriageways where 'handling is not needed'. However, the bulk of the network is the older type of network where this handling thing does matter. In fact, we kind of chuckle with embarrassment at the sight of U.S. cars and their crude engineering. In a narrow country lane when the 325i meets the U.S. soccer-mom lard-mobile, you know that the guy in the 325i will be the one flicking into reverse and backing up to a passing point.
If your experience with driving is sitting in traffic or driving around suburban streets, you're right, handling (mostly) doesn't matter.
[1] https://s-media-cache-ak0.pinimg.com/originals/ca/e2/45/cae2...
But I admit I might feel differently if I had to navigate roads like that on a daily basis.