No, clearly it's not Free as in FSF, but 'free as in beer' is the only thing that separates it from H.264, it's definitely the most accurate description I've seen yet.
Stallman himself agreed that it was a good thing for Xiph to relicense their libraries from the LGPL to a permissive free software license (http://lists.xiph.org/pipermail/icecast-dev/2001-February/00...):
In response to the change of license, Richard Stallman of the Free Software Foundation says, "I agree. It is wise to make some of the Ogg Vorbis code available for use in proprietary software, so that commercial companies doing proprietary software will use it, and help Vorbis succeed in competition with other formats that would be restricted against our use."