Yes I was thinking about that too, it's more like using the SNES itself as a video adapter.
But even beyond that, this isn't the case - the core logic for scene creation runs on the SNES and then the instruction opcodes are shipped to the expansion chip. So it's a GPU for the SNES, and a perfect analog to the SuperFX chip from which it takes its name.
A far cry from simply "using the SNES as a video adapter" as the full SNES hardware is still running the "game logic" to accept input and set up the scene.
It's all about expectations...
The inherent limitations of an SNES cartridge are the power consumption (although some cartridges actually included batteries) and the memory bandwidth (it's a 16 bit system so the cartridges have 16 pins).