A website relying on a single service to serve as a sole identity provider seems replete with risks. That sounds remarkably similar to Microsoft's Passport, an idea that was a non-starter for everyone except Microsoft.
But even with the current implementation it is possible to validate the assertion in your server without contacting browserid.org, but AFAIK nobody did it yet.
(I tried to find a reference for that, but they took it down from their Wiki. It used to say "You may choose to validate assertions on your own server": https://github.com/mozilla/browserid/wiki/How-to-Use-Browser... - the verification code is still online, though)
Faaborg's mockups are awesome, but they're well over a year old and not really guiding our current work. Still, definitely take a look at them, if nothing else than for the cool way of doing mockups!
You can definitely do your own verification without having us in the loop, but we'd urge you to hold off until this fall; the IETF is still standardizing some of the data formats we're using, so the exact serialization of keys and assertions might change between now and then. By using our verifier, you're certain to be up to date.
The easiest way to verify-it-yourself is to just run our code locally; it's all open source, after all :) https://github.com/mozilla/browserid
(PS: The docs are now on MDN, instead of the GitHub wiki: https://developer.mozilla.org/en/BrowserID )