Some people simply aren't comfortable with it.
Currently sound Rust code does not depend on the value of uninitialized memory whatsoever. Adding `freeze` means that it can. A vulnerability similar to heartbleed to expose secrets from free'd memory is impossible in sound Rust code without `freeze`, but theoretically possible with `freeze`.
Whether you consider this a realistic issue or not likely determines your stance on `freeze`. I personally don't think it's a big deal and have several algorithms which are fundamentally being slowed down by the lack of `freeze`, so I'd love it if we added it.