The concern with the Intel ME is that it has a native network adapter. You can bet efforts are currently underway to discover how to exploit the ME remotely. THAT'S when things get scary.
Your paranoia is not unjustified. Personally, I am nervous that some of my systems have the ME. When attention turned to it about a year ago, i knew it would only be a matter of time before someone broke into it.