The hardest part of building this business is to keep motivated for a relatively long period of time. I'm still early in this startup journey. This is only the 2nd year of me working full-time on Listen Notes.
I think it would be helpful to surround yourself with like-minded people (online or offline) -- we are social animals. Indie Hackers is pretty good: https://www.indiehackers.com/
I live in San Francisco and I used to work for companies, so at least I can often hang out with some friends/former coworkers who are doing startups or working in tiny startups.
In my coworker space, people from different companies rarely talk to each other...