However, there has to be a line where a remote control operation could interact with the national airspace system and the regulated operators therein. Not more than 500 feet and away from airports sounds like a good limit to me, adding on that the drone operator must maintain clearance from any aircraft that may exist in that area, such as helicopters.
I think the key change that is spooking the FAA is that drones may be operated (from a technical perspective) outside of direct view of the operator. "Traditional" model aircraft weren't generally taken far from the field they launched from; but a drone with a camera and transmitter can go much further.
Probably everyone has seen the video of the drone that was flown through a fireworks display. A very cool video indeed, but, also a good example of the FAA's concerns: if the drone is outside of view of the operator, and running at altitude, and not carrying a transponder, lights, or other equipment mandated to certified aircraft, then how is clearance from other aircraft ensured?