But what if I want to encrypt "my secret" with "password1234"?. You can't require everybody to memorize exactly 32 bit passwords
A cryptographic key (unlike a password) needs to be exactly a certain length, and ideally be completely random (if not, at least indistinguishable from random): a password might be exactly that length, but it will _not_ be completely random (heck, if you can type it then certain bit patterns simply won't occur …).
The fact that '012345789ABCDEF012345789ABCDEF' and unhex('cd8fa5ecf22379c429f56fdaae511144f9b7704247a244859b943299eb57fb1d') have the same length in bytes is coincidental; they really are two very different types.
Furthermore, use a KDF to stretch your password into a key suitable for encryption.