Binary package management in FreeBSD isn't well-advertised, but it's definitely there:

  # portupgrade -rvP --batch foo-1.2.3

This will upgrade installed package foo and anything that depends on it using binary packages from the mirror site, falling back on a build from source.

Installing a new binary package:

  # portupgrade -vNP bar/baz

If you're using portaudit to stay on top of vulnerabilities (it's part of the default install these days), here's a nice way to upgrade everything:

  # portaudit -a | sed -ne 's/^Affected package: //p' | sort -u | \
     xargs portupgrade -rvP --batch

I'm like you, for some vulnerabilities I don't want to upgrade a ton of stuff. FreeBSD is perfect for this though, you can just pop into the port directory, "make extract," apply the patch, and "make reinstall." You can avoid full rebuilds by always passing "-W" to portupgrade whenever you install / upgrade stuff.