(Of course, in practice people made a hash of it (as they always do) and it often became mobile-only design, with users of larger screens getting a comically bad experience that was manifestly designed for tiny screens; but that’s not the idea of mobile-first.)
I’d say it’s generally a sound philosophy, even though at larger sizes you may desire to recompose the UI quite substantially. It’s generally easier and/or safer to design for small screens and reformat or add extras for larger screens, rather than to design for large screens and reformat or remove for smaller screens.
But people definitely regularly don’t go small enough in their idea of “mobile”. I decided some years ago that I would target 300px as my baseline width, and I feel that’s served me well. (And if convenient, I make it work past 260px.) 300px is a bit smaller than anything mainstream, and is thus pretty safe. (I haven’t often done much that needs a baseline height, but I’d use 300px and/or 450px there, considering both portrait and landscape usage as necessary or applicable.)