We probably could delay it until the setting is enabled. I wasn't on the team when that decision was made, but I would imagine it's because a) latency (we want the feature to be enabled right away when you turn it on), and b) just the way it happened and nobody really thought much about it at the time.

The fact is that an end user should not care if software downloads a "binary blob" without running it. This is functionally equivalent to downloading anything from the Internet, a JPG file for example. Chromium downloads a bunch of things on startup, and nobody seems to mind. Just because hotword.nexe happens to be an executable blob doesn't really make a difference.