It's a small world.

Your hiring manager, the people you interviewed with (6-8 people probably?), and some of the management at company A will know you rescinded your signed offer.

What are the chances that one of those folks ends up working a company with a job you're applying for 4 or 5 years from now when your application comes through?