Unless things have changed with the iPhone 4, the only case in which downgrade is not possible is if the upgrade modifies the baseband firmware. At least that has been my experience. I've downgraded iOS on my phone a number of times, most recently from the latest beta of 4.1 to 4.0.2.

You may argue otherwise, but from where I stand, preventing downgrading and thus preventing jailbreaking has been a (fortunate for Apple?) side effect, rather than a goal.