> What happens when someone publishes some malware application using their wrapped and noterized installer
The installer application simply installs the app. It doesn't in itself run any of the app's code. The installed app doesn't need to be codesigned and notarized like it would if you had just downloaded it from in your web browser.
This works fine for many cases. A limitation is that apps built in this way can't be submitted to the app store. For that you would use jpackage or similar. But in most cases, this strategy is fine - and even better since it includes things like auto-updates.