Is it actually a substantial expense? The TLD itself only has to publish the nameserver records, which generally have a TTL of about a day. A DNS response is a few hundred bytes. Big DNS providers like Google and Cloudflare would make requests for every actively used domain every day, but then cache them. Smaller providers wouldn't cache as well but also wouldn't each request every domain every day. For e.g. a million personal domains, ballpark estimate is somewhere in the few TB a month of traffic. Maybe a little over personal hobby project money but definitely not outrageous for a small non-profit organization.
> How do you plan to tell the difference between a parked/squatted domain and one in legitimate use but offering no public-facing services?
This is the easy one. Squatters buy domains because they want to sell them. To sell them they have to make it publicly known to prospective buyers that the domain is available for sale. So then if anyone lists the domain for sale anywhere, you make them prove that they own it (which any actual buyer would also have to do in order to not get scammed) and when they do the domain is forfeit.
It's kind of sad that we don't do that for all domains. Domain squatters can go to hell.