I think you are going about this incorrectly. Do not prioritize on commonality as your user count remains small. Instead prioritize on severity then carefully examine each defect as they come in. You are either willing to work the defect or you aren't. There is no harm in marking an issue as out of scope or won't fix. Just be clear and direct with your users that you have plans for this application and those plans only move in one direction.