It's hard to remember that period well and put things into context, but if you were there, honestly, Google Video was pretty good. In the very early days Google Video vs YouTube
could have been a toss-up. There was nothing particularly compelling about the UX on YouTube (it got much better later). Google Video wasn't amazing UX, but technically it was fine. And had the advantage of getting prime real estate in search results.
YouTube's early success I suspect had a lot more to do with them paying fast and loose with enforcing IP rules. It wasn't until a bit after the Google acquisition that they started getting more serious about it.
EDIT: also remember this is an era when Google was actually doing quite well shipping rather geeky and minimalistic invented-by-engineer consumer-facing things. There was no reason to assume they wouldn't do well with Google Video. They had just recently had a lot of success with Gmail. Gchat was a thing. Their calendar was good. They were making some ok stuff.