So far, the biggest problem is simply learning how to find music again. Spotify was too convenient for discovery.
But, I've been happily streaming my collection for a couple of months now, and removing Spotify and the radio has gotten rid of most of the last remaining advertising vectors in my life. It's nice. I very much enjoy knowing that I own and control my media and the data about my usage of it. There will never be ads or a creepy business model. If something goes wrong and knocks the server offline, I have my music cached on my phone.
I wish this were more accessible to everyone. Setting up a public-facing server, or even a private one with a VPN tunnel is too much hassle for the average person. It's so very worth it though. Owning your own services is so liberating