"Basically what you do is to put "fences" around dangerous areas as a manager and don't confront them unless it is "too much" - where too much depends on a situation."
In your example, putting a fence was apparently an efficient way to mitigate the risk. Just like in risk management, you need to establish your risk tolerance and know in advance what type of behavior you are willing to accept.
Once you have identified the risk, discover the root cause, and then you can choose whether to avoid, accept, transfer or mitigate the risk.
Some risk often remains, and often new risks come as a result of controlling the original risk (https://teamsuccess.io/risk). For example, firing a liked employee might lead to others leaving too.
Lots of companies and teams don't establish a risk tolerance / risk threshold, which leads to managing the situation poorly.