The issue isn't leaving, but abdicating your responsibilities. This is clearly what you care about as well, when you say "I can't ditch them." While there certainly would be something to be said for the "meaning" of what you'd pitched if you leave, the key here is making sure that the actions you take do not negatively influence everyone else there and the people you brought into this venture.

I think the question to be asking is "What steps can I take to transition my responsibilities to other people without negatively affecting the company and my colleagues?" If you can find an answer to that question--hiring someone who you groom to replace you, making the business more institutionalized and self-sustaining so that you don't need 14-hour days, etc.--you'll be closer to getting there.

It's pretty incredible how many people I see who start their own companies hoping for freedom, and find the opposite. Responsibility often means less freedom, and having clients and employees beholden to you means more responsibility. In some ways, being employed can mean being more free. There was a great article on HN recently pointing out that the true level of freedom for entrepreneurs is likely to be found not at founding your own company, but specifically at founding your own moderately successful company--once great success comes, you have much more responsibility.