Having been in this situation more than once (I caught the startup bug), I can tell you that there is no good way to leave an unsuccessful startup.

What I learned is, my time is mine to protect and my skills are mine to dedicate where I see fit. Noone can oblige you to spend these things on what you don't want to.

People will always blame others for their lack of application. If they were true entrepreneurs, they will find a replacement for you.

But you are unique that you want to leave what is really a pet project (I've been in those too many times) and build a real business. You should!

Don't worry about what others will think, you can't control that.