I think there are some hooks in OSX/Cocoa for doing this, although I don't claim to actually understand it.

There is the SIMBL system which people used to write native plugins for Safari, I guess that's probably stopped since it has an actual extension framework now.

http://www.culater.net/software/SIMBL/SIMBL.php

Then there's TotalFinder, which uses in-memory patching to add tabs and all sorts of other leet stuff into Finder. I highly recommend it. In their docs they say it works similarly to SIMBL.

http://totalfinder.binaryage.com/