So what I did:

Trying UBports. Got an used phone from this list: https://devices.ubuntu-touch.io (in my case, a Google Nexus 5, bought it 4 years ago, I think - I'd get something newer today) and get a prepaid sim-card without internet.

I do my updates and app installations at home when it connects to my router. Without internet, I never use the browser, don't have chat/social media apps installed. Maps work offline (download the regions you need at home beforehand). Music is stored on the device as well.

If you need an overview of available apps, this is the store available on the phone: https://open-store.io

For app development (I don't do that though) I think this is a good starting point: https://docs.ubports.com/en/latest/appdev/ and probably https://forums.ubports.com/category/35/app-development

I both loved and hated J2ME. It kinda had what you needed though, apart from the often "openly" idiotic device implementations that were kind of the norm. Only Sony Ericsson had a great one (fast, high code quality, sane security policies/prompts). Nokia came at second place with medium performance, high code quality and outright user-hostile security policies/prompts.

Consider getting a PinePhone. The A64 CPU on there is so old that web browsing is painful, but you can still use it as a music player, check email, and hack together whatever little apps you want with your Linux framework of choice.

The sweet spot for me for a while was the iPod Touch. Could load stuff into Instapaper to read on the subway, take notes, write emails, and otherwise be left alone by the Net so I could peoplewatch instead.

Now I think the main excuse to have a phone is the camera, and I'm looking at pocketable digital cameras on eBay.