Having faced the same situation in the past, I encourage you to simply release it. No need to spend months upon months improving the backend while no one is using the damn thing.
How long has the refactor been on your mind? Has it been at least a month or two? Has your mental picture of the rebuild been consistent, or does it change week-to-week?
Here's my advice to get you going: Write down your general refactor plan in detail. Next, create a todo list / timeline that gets you to production deployment as quickly as possible based on your current codebase. Focus on other things a bit. Launch, marketing, etc.
Once this thing has been out in the wild for a bit, reassess that refactoring plan you wrote down and see if it still makes sense based on your application's performance - both as a product, and as a piece of software.