The annoying thing, of course, was that you couldn't plug in a rumble pak and a memory card at the same time. There were third-party options available, but third-party memory cards had a bad reputation.
The Dreamcast solved this by having two slots. The VMUs were insanely cool at the time, and honestly still are. Some games used them in cool ways, such as Resident Evil showing your health.
Sadly, it wasn't for gaming. It was part of a study into the limitations of how much information humans can absorb at once, with the haptic feedback being tested as yet another input when there was a lot of auditory and visual input. I joked they should just use smell, but I don't think they wanted to subject the undergrad research subjects to weird smells.