I am really surprised OpenAI didn't think about this and deliberately try to make these plugins not work with third party engines.

For example, they could have required the schema file be uploaded to them in your account. That way the majority of plugins wouldn't have publically accessible schemas.