The SAFE Act as originally introduced in the House of Representatives was designed to be generally pro-crypto by relaxing export controls. But as it made its way through the various committees, the anti-crypto forces got their hands on it and turned it on its head. It became a ban-non-backdoored-crypto bill instead.
More precisely, in 1997, a House committee approved a ban on domestic encryption without backdoors for .gov access. Here's an excerpt from the amended anti-crypto version of the SAFE Act:
"After January 31, 2000, it shall be unlawful for any person to manufacture for distribution, distribute, or import encryption products intended for sale or use in the United States, unless that product [...] permits immediate decryption of the encrypted data..."
Here's how one of the anti-crypto politicos, Rep. Bill McCollum, who went on to be Florida's attorney general, justified it while debating the House Judiciary version of that bill:
"Because this bill will promote greater use of stronger encryption, law enforcement may not be able to gather evidence that it can use to investigate and prosecute cases. Imagine a situation where the police with a search warrant seize the computer of a terrorist but cannot decrypt the list of people and places that he intends to strike next. Or the situation where the police seize the computer of a purveyor of child pornography but cannot decrypt the files to download the images to prosecute him." http://www.techlawjournal.com/cong106/encrypt/19990324mcc.ht...
So yes, you're right that sec. 2804 in one version of SAFE eliminates mandated key escrow. But other versions, including the one approved by that House committee in 1997, went exactly in the opposite direction.