No. You release one .apk file (the Android equivalent of .app). Within a manifest inside it, you can specify a "target" release API version and a "minimum" version. No one running a device below the minimum can see or install your apk. The target one is more of a suggestion to the Dalvik VM with how to approach running your application (if it know you're targeting the latest release, it can expect you to use some new features and adjust accordingly, etc).

Just like with web sites, you progressively enhance an Android app during development using Java's reflection, and/or gracefully degrade by bundling up back-ported modules and UI components in your apk. If you absolutely can't get around a particular function not being present in an old version of the API, then you just don't have that feature even visible to the end-user.

The current recommendation is to set your min level to 7 (which is Android 2.1), and as of a few days ago, to set your target to 14 (Android 4.0, or Ice Cream Sandwich). According to raw statistics of devices active and in the wild, this hits 97% of all potential Android customers. Fragmentation used to be an issue back when supporting the Android 1.x line of APIs was still necessary. Now it's nice and streamlined, and I barely notice writing for compatibility any more.

And if you really can't get around some problems, the Market now lets you upload multiple apks for a single "release" of your app, each targeted towards different ranges of API levels.