I’m not a big fan of this less features ethos. This feature anemic set of software that gets put out more often than not drives me bonkers. It’s as if they don’t want power users.
It's a really strange (and wholly unnecessary) cycle.
BUT apple specifically has ExtensionKit, which is a library for arbitrary extensions in arbitrary apps. Of course, the host-app needs to support extensions and build it in. So your theorizing about a way for any app to have a plugin, its there. Many system apps in OSes have this feature, and many more apps (eg vscode) support it.
Realistically, web-apps are almost what you want too. Web-apps allow for pretty arbitrary extensions via browser plugins, which can modify the actual UI of the webpage, and make API calls to the backend (by acting as the JavaScript on the page). WebApps are basically open source though (or source available) by nature of how browsers work.