How would you do certificate pinning if you don't control the clients?

My understanding is that certificate pinning is only possible if you control the clients, in which case you can embed which certificates are allowed directly in the client and bypass the whole web PKI.

In a situation with general-purpose clients connecting, how would they know which certificates are meant to be allowed? That's what the web PKI is used for.

Of course, if you do provide your own clients, this just moves the problem further up the chain - in this case the place where customers would download the custom client software would be compromised and a malicious client served instead.