One argument is that cocoa farming is a driver of deforestation in some regions [0]. Besides the environmental angle, a lot of cocoa farming is child labor [1].

[0] https://e360.yale.edu/features/the-real-price-of-a-chocolate...

[1] https://www.washingtonpost.com/business/2020/10/19/million-c...