You can install applications on Android devices by logging in through the web interface and buying them. The next time the Android Market app syncs with the service it will see it has been bought and will download automatically.

So not just anyone can install things to your phone, just whoever has access to the Google account associated with your phone.