Prior to FreeBSD 10, I'm paraphrasing but essentially it would pretty much not swap anonymous pages until memory pressure got really high -- it would evict clean disk pages and try to clean dirty disk pages first. But if the pressure continued, it would start marking anonymous pages and eventually swap those out too. A big problem was there was a massive slowdown when it hit the threshold; tons of pages would be marked as inactive, so using them in the process triggers a page fault (and they then get marked active), and all the page faults and page table activity slows things way down.
In FreeBSD 10, a patch from NetApp changed the page scanning behavior so it was always running. This is good, because you don't get that massive slowdown while the pages are determined to be active or inactive; instead the kernel always has a much better idea of which pages are used; but it's also not great because some of the inactive anonymous pages are going to get paged out, and it also makes memory accounting trickier. In FreeBSD 9, you could look at the active memory stat on a system that wasn't at the brink and figure that was pretty much the amount of ram you needed -- so if it was growing over time, you might have a memory leak, or real growth. In FreeBSD 10, it's harder, because your programs may need things loaded but not access them for some time, and the pages could get marked inactive, so you really have to guess; and swap usage isn't a great metric either because those idle pages might get swapped out. You can set the vm.defer_swapspace_pageouts sysctl to get something similar to old swap behavior, but you still lost the memory usage gauge.
OOM behavior seems better, but also not necessarily great. Most of my experience is from running Erlang servers with very little other stuff going on, so it might not apply to a more conventional server with lots of random things, and probably not to a desktop either. Overcommit is disabled by default (and i think it may be substantially different than Linux overcommit anyway), and mostly what happens in my experience is the big allocator (beam.smp for us) gets its allocations failed when you run out of swap and then it crashes and everything is in an ok state. Every once in a while, something else will get failed allocates and die before the big allocator, but then the big allocator still dies; this wasn't great because sometimes it would be an important but small daemon like ntpd, when this happened we'd often reboot to get back to a known good state. There's also an OOM killer that can be triggered depending on exactly how things go down -- it just kills the biggest process; for us, that's the right thing, if the VM is eating all the ram, killing off gettys isn't going to do any good.
However --- we saw several times that the kernel would just hang. Unfortunately, when we were seeing this often, I hadn't figured out the kernel debugger, and it was usually in the middle of an incident anyway, so we'd reboot and go on with life. A hung kernel is better than a thrashing kernel, but only a little bit.