They would all use the same reserved "name". It's possible to atomically create or open a SHM segment with a given "name". Then have a header at the start of the segment with critical metadata like signature, version, etc. Barf unconditionally if the metadata is found to be invalid (i.e., some other process deliberately used the name to sabotage sqlite [1]), or, in addition, have an option to not use SHM.
[1] The situation is not different from some process sabotaging another by deleting or corrupting well-known files.
As a practical matter, you really have to work hard to get an application to use two different copies of SQLite at the same time - all the while avoiding symbol collisions on link. You can do it, but it takes some work. And then on top of that your application has to decide to open two or more connections to the same database file, using different copies of SQLite in each case.
The more common failure mode I've seen is a library bundling and inlining an old version of a database API in a way that means it overrides the shared object I was expecting my code to link to, and alarums and excursions resulting thereby.
(I don't imagine SQLite would -break- in that situation, but I doubt my application code would be any less unhappy than in the cases of that problem I've encountered in the wild)
It's easy: create DLL/.so A with statically linked sqlite. create DLL/.so B with statically linked sqlite. Make a program using both DLLs.