> Jefferson had planted opium poppies in his medicinal garden, and opium poppies are now deemed illegal
Growing with the intent to produce opium is illegal. By continuing his tradition in the museum, it was deemed a continuation of production for drugs and so probably came under scrutiny. I'm sure they could regrow a garden for decorative purposes if they actually did that and were clear about it.