For users, app stores give safety as others mention.
For app creators, app stores give access to the walled garden.
> What would I lose if I'd simply provide my app as a download on a website, like some app creators do?
Very few non-technical users install apps from random websites, or even know how to do that, so you'd likely lose your install base. If the app is circumventing Google ads, then yeah, the Google app store may not work. If it's a general interest app then spend the effort to post to the app stores.