You can read their terms here: https://developers.google.com/youtube/terms
Specifically they say it is prohibited to "separate, isolate, or modify the audio or video components of any YouTube audiovisual content made available through the YouTube API;"
and it is prohibited to
"promote separately the audio or video components of any YouTube audiovisual content made available through the YouTube API;"
Like I said, I'm not a lawyer so my interpretation could be wrong.
I think in a nutshell they want you displaying the video because they have their own ads in the videos, and just streaming the audio cuts into their revenue.
Hate to be a wet blanket but I recently built a service using the YouTube API (http://chartbrain.com) and so the terms were fairly fresh in my mind.
I think other services that used only the audio from YouTube have run into trouble - songza comes to mind, they were one of the first sites to do it. http://www.wired.com/listening_post/2007/11/songza-turns-yo/