Unfortunately our neo4j setup as it stands really doesn't scale. At an application-level we're built error-handling to gracefully handle the frequent outages we experience running it.
These are probably our fault, rather than a failing in the product, but I've wanted to replace the current neo4j-based suggestion algo with a new one that uses batching/clustering for a while. As soon as I get a chance, I would like to remove the dependency we have on neo4j.
Additionally, Neo4j does support clustering with their Enterprise edition. It also has much better tooling (better metrics/logging, backups, etc.). It is AGPL licensed, so there's no reason not to use it really.
Since it uses Gremlin, you should not have that big a problem in porting.
If you want pure Open Source then Titan might be a good fit for Gitter. http://titan.thinkaurelius.com/
Thanks for your recommendations. I'll definitely check them out.
That's why in recommendations, I usually filter them out b degree (which is a constant value read).
MATCH (r:Room) WHERE size( (r)<-[:MEMBER]-() ) < 5000) ...
You're also not using directions in your queries, which will also help.
I looked at your other queries, they can also be improved easily, if you want to I can help you do that. Just need some sample data or a read-only access to the db. Just ping me at michael@neo4j.com.
I'm not going to be changing anything around the suggestions any time soon, but who should I get in touch with when I do?