There’s only a few states (CA and few others) where the employer can’t claim IP developed on your own time and equipment. The other states absolutely can.
Yes if you invent the next google in your spare time you could be in trouble. For this it’s best to get funding. At the lifestyle income level, your employer doesn’t have the time or money to waste on your passive income stream.
Obviously don’t be cavalier about it, but this will just get you fired not sued.