We use it via FBML and JavaScript SDK and the slow down seems to be minimal. We load their script right before </body>.

Biggest annoyance is that it doesn't degrade very gracefully when FB has availability problems, you get a white square in the page in Firefox.