I came to a similar realization recently. I wanted a music player I could put in my seven year olds room for calming music and sleep stories. Something basic like the iPod shuffle was what I had in mind, but in 2024 I'd need to buy an iPhone or iPad if I went with Apple. Not having a screen was a key feature. I eventually bought a half dozen $2 iPod shuffle knock-off on eBay (there's several options in this price range). I needed to buy a microSD card separately though. So far they have worked great.
I am also using a simple iPod shuffle knockoff with bluetooth when running, works great!