I asked our allergist about oral immunotherapy for my daughter and he cited a study that found that avoidance was more effective in preventing severe reactions.
Oral immunotherapy seems to be very effective but sticking with it is nontrivially hard.
I wouldn’t tell you to not trust the advice of your doctor. But I would tell you that there is an always argument both ways for something like this. A different doctor may feel differently, and sometimes doctors change their minds when given a bit of pushback. It might be worth asking about the possibility of oral immunotherapy in addition to avoidance, rather than as a substitute for it.