Few people know that when Uber started and the first $1M was raised, the apps were built by contractors. The app was bad, the code terrible - but even with a bad app, customers used it over taxis, that didn't even have a bad app. The business took off, the next round of funding came, as did the first few full-time engineers.
The first thing that the full-timers did was throw away the mess of a code, and rewrite the app. However, moving fast was still more important than quality. Launching in a new city needed to be done in a few weeks - if the ops team could mobilize a whole city in this time, engineering was expected to move fast as well. So while generally, forward-looking decisions were made, still, many-many shortcuts were added, most notably "The Ping", which was the backend sending over all state data to the client in a massive JSON object, ever 10 seconds. This was to speed up development, not having to make backward-compatible state changes all the time. It's something I'd cringe over today, but it did help moving fast, at the expense of loose contracts and lots of bandwidth usage that could have been avoided.
As the business proved to be successful, in year 3 or 4, reliability and quality started to be more of a focus: things like tests, listing, architecture, rollout best practices, and so on. A big push happened when in year 4 or 5 (I can't remember exactly) a sloppy change almost took down all of Uber's core systems at rushour. But for the first few years, quality took a relatively back seat. Was it worth it? I'd definitely say so. As another commenter noted, the customers of a startup do not buy code quality: they buy something that meets their needs and is good enough.
When a startup becomes wildly successful, you'll have the funds to pay off tech debt. Until then just make sure it doesn't suffocate you - otherwise pile it on, and move fast.