Not necessarily. Sony has joysticks that can snap in and out of the advanced controllers. It wouldn't be hard at all to design a backing circuit board that supports this behind the trim. Switches aren't exactly delicate parts either so it's conceivable that a cheaper system could use auto shops to solder in a new switch into the board.
You could also have a simple multiplexed interface board near the head unit and the switches could use simple two wire connections back to that board. Or the head unit could just have this built into it. Or you could design and use a HID like protocol so different interface adapters with different capabilities could be plugged and unplugged from the system.
Is this worth the cost? Short term probably not but long term you might be able to make these accessories much more generic and so reusing them in newer designs might actually lead to good savings. Plus you'd spawn an active third party market for these parts.
If you want the car to be fully customer configurable, you basically need a custom dashboard for every single car. You also need to think about what happens when the customer does an upgrade.