The obvious big issue is maintaining roughly the same salary level, but you’d be surprised how much you can tighten your belt if a making a big jump down. A non-obvious negative is getting used to loss of status. That hurt a bit initially, but I soon found that the novelty of re-inventing myself in a new domain was massively invigorating, plus I was suddenly working with very different (and much younger) colleagues. So I decided to shut up about the old job from day one and never mentioned it unless asked (no-one cared anyway).
But be aware of turning a hobby into a job though. I got into dev as just a hobby initially. Then it became a paying part-time gig when on leave, which eventually lead to a job offer via someone I knew in the business. You’ll soon find that doing your hobby for a living cools your enthusiasm for your hobby, especially when dealing with difficult customers, bosses, or ridiculous deadlines. That said, I’m really pleased I made the jump and don’t ever have to wonder “what if?”.