The spookiness, at least for Americans, came like so:
1. Gilded Age upper classes built the fanciest mansions they could afford, in the Neo-Gothic style which was fashionable at the time
2. Like the English country houses (see https://en.wikipedia.org/wiki/Destruction_of_country_houses_...), eventually these rich owners couldn't afford the upkeep of these massively oversized and ornate dwellings. And nobody would buy them. So they moved out and left the mansions to become ruins
3. Now lots of people know about the old abandoned mansion on the hill. Gothic! Spooky! That includes Charles Addams, who starts making jolly cartoons in the New Yorker about the odd family that live in a big spooky mansion, and it includes Alfred Hitchcock who thinks a run-down mansion is a great setting for Psycho