My hunch is that a few million objects is pretty tiny these days - you could probably write a script to import all of them into PostgreSQL in a few minutes and try it out yourself to see how it behaves.
https://gist.github.com/simonw/c16ce01244760e186a3a0aa3fee04...
Then I ran that query again and it seems to return results in about 80ms:
https://lite.datasette.io/?sql=https://gist.github.com/simon...