For traditionally-shaped planes (delta wings work differently) there's a fairly straightforward tradeoff between big wings to increase lift at low speeds and small wings to reduce drag at high speed. Flaps help but there's a limit to how much you can change the shape of a wing in flight. So if you build a plane to go very fast, you tend to end up with a plane that has a high minimum speed as well, which makes take off and especially landing difficult.
To simplify, are you basically saying that with more unstable wing configurations, at slower speeds the plane simply wants to fall out of the sky rather than glide? Is that sensible to imagine for this topic?
A car has good stability in turns: it wants to go straight, if there's a bump or you jerk the wheel a bit it still goes straight. If you apply a control input and get a turn, it will let you maintain the turn but it also wants to unwind and go straight. This is stability but not aerodynamic stability.
The reason one might want a less-stable aircraft is that stability is the opposite of manoeuverability. Super-stable aircraft like training gliders really resist turning and can feel too much like hard work.
As another commenter mentioned the wings don't contribute much to stability. The two big fins at the back (the stabilizers) are the main source.
(disclaimer: I fly fixed-wing acft and gliders, barely, and am no physicist.)