Passion project? It is the sole source of his income.
You're not an employee for somebody else (correct? It's your own project) - it's not a salaried or hourly position where somebody else is currently signing your paycheck. You obviously don't hate it (I hope? If so, not sure why you wouldn't have canned it yet)
Thus, this is what most people would refer to as a passion project. You may have a different meaning of that term in your own mind, but I think, on average, most people would refer to what you're doing as such, and not to be malicious in doing so. I wouldn't let it hurt you, I'm sure that's likely not a friend's intention.
FWIW I have found that three things help me deal with this sensitivity:
1) Take it as motivation to be more outwardly "professional" about my art. Improve the website, be more active on social media, try harder to exhibit, even (ugh) sell things.
2) Remember that most people are just saying that out of ignorance: they don't have a mental model for anything outside of "work for the Man" or "mess around in free time."
3) Have examples ready if someone needs an explanation, and also as inspiration for yourself. For instance: was William Carlos Williams[0] a "hobby" poet?
[0]: https://www.poetryfoundation.org/poets/william-carlos-willia...
In the case of Dwarf Fortress, I would say for him it is "the sole project he's been working on for 20 years now, which takes up $large_number of hours per week, and which has been sustaining his life now thanks to a fan base which have proven themselves reliable contributors of financial support. And he really enjoys writing the code too"
This is spot on. If only you would articulate my feelings for me all the time, that would be great!
Even if I deployed his code tonight, there's no guarantee users would switch over to my LeeChess clone :)
In the case of Dwarf Fortress it's just "download my exe and please donate if you like it!"