"Getting back to the sci-fi, I realized that the crystal swapping could just be the high level interface to all the complex code below. "
Personally I find it a bit hilarious when I think about it too much. We can fit a terabyte on something that fits on our thumbnail comfortably, and we can provide enough computation power to run most any plausible control system comfortably in a phone form factor, but the superadvanced aliens way smarter than us and millennia if not more beyond us need massive crystals.
I suppose we can hypothesize that some variant of "hyperspace" requires vast computational powers to work at all, and maybe some of the other advanced stuff does (e.g. one imagines the holodeck or anything even resembling it would possibly require more computing power than our entire world has), but it's hard to imagine "life support" requires some massive crystal to run.
Science fiction has always had a complicated relationship with computers. The exponential power increases of so many decades is just hard to deal with. Even if you understand that exponential power increases are occurring, that doesn't mean you can correctly understand what that means in practice.
(In the alien's defense, one thing they may legitimately have on us, and for which unspecified "crystals" clearly made out of a very hard and durable substance may even make some sense, is longevity. A crystal that "merely" stored a few gigabytes and had the computation power of a modern cell phone, but that could be accidentally left behind in someone's grave for 20,000 years and work just as well as the day it was made would have a lot of legitimate uses, even if they had the capability of making things with a lot more power that wouldn't last as long. But even then I don't see swapping these things around like they're a fancy puzzle from Rubiks or you're trying to solve an ancient Sudoku as being a likely use case.)