Not signing the contract is one thing. Misrepresenting your position is something else. If you gave the impression that you agreed to the terms provided, then it's possible you could be held to those terms.
A contract isn't defined by a signature. A contract is defined by a "meeting of the minds." The signature is commonly-accepted evidence that such a thing occurred, but it's not essential.
By pocketing the contract rather than pointing it out to them, you've arguably violated the "meeting of the minds." The company was under the impression that was part of the agreement. You can dither about "due diligence" and say it was there responsibility to check, but in some cases that can be seen as an intentional misrepresentation on your part, for which there are consequences.
I will concede that this is unlikely to come back to bite you in practice. But others should be aware that it is technically playing with fire. And personally, I consider it unethical.