1. Natal kicks. The process of forming a black hole is fairly violent, and could involve a supernova in many cases. If the supernova is asymmetric, this will result in a kick on the black hole. Even if there is no supernova, an asymmetric infall process could produce an asymmetric burst of gravitational waves. Since gravitational waves carry momentum, this will also result in a kick on the new black hole. But it's generally thought that black hole natal kicks are weaker than neutron star natal kicks. The received wisdom seems to be that natal kicks will remove some black holes from a globular cluster, but probably not most.

2. Few-body interactions. A general principle of dynamical interactions (called Heggie's Law) is that if a single star interacts with a binary system, it is likely that the binary will become tighter if its orbital velocity is greater than the velocity of the incoming star, and wider if its orbital velocity is less than the velocity of the incoming star. The BH-BH binaries that would form in the cores of globular clusters would be relatively tight, so almost all interactions they have with other black holes will result in them getting tighter and giving more kinetic energy to the interloping black hole. (Or there may be an exchange and the interloping black hole will take the place of one of the black holes in the binary and kick out one of the original black holes.)

Back when I was in grad school I made a visualization of a binary-binary interaction, which you can watch here: https://www.youtube.com/watch?v=Y3nPOQXl6co