If you take your car on the track there's a very high probability it will get seriously fucked up. Unless you're cruising around casually you're going to be taking chances, and when you take chances you'll eventually go off the road, grind into something, or worse.
Fixing your Porsche is expensive. Fixing your Ferrari is stupidly expensive.
If you want to have fun on the track take out a car you can afford to wreck. A Porsche with a track setup you can really let loose on is going to be way more fun than a Ferrari you need to be gentle with.
There aren't many people that have enough money to not worry about wrecking their Ferrari, and fewer still that have time to bomb around on a track in one.
Plus, if you run your tuned up track Porsche into the wall and it's a write-off, hey, shit happens. If you do that with a Bugatti or an Enzo you're just being a dick. Those sorts of cars are rare and deserve some respect.
There's also another factor: After-market modifications for cars more commonly made are easier to source. You can mod an Audi or Porsche easily, it just takes money, but doing the same with a Ferrari is significantly more complicated, you'll probably need to have a lot of things custom fabricated.