The point is that they provide the tool for clients to decrypt the URL. And youtube-dl is a YouTube client, just like Chrome or Firefox is. If you have a key to a locked house, unlocking the door doesn't mean that you "circumvented the door" and went in through a window. I really don't know how else to put it.
I might be missing something, but I'm not sure what's the caching thing is about. YouTube tells you under what URL you can access the specific video or audio file, you download it and that's it. Whether the downloaded data is saved to a file or temporarily stored in RAM or whatever is up to the client. And the way everything is structured right now, I really can't see any other way they could do it without creating some kind of proprietary YouTube client.