Tomato are closely related to tobacco IIRC (as well as potato).

There might be traces of nicotine in the plant, which is a pesticide.

This actually also might give some fuel as to why people considered it dangerous.

EDIT after googling: Yes tomato contains nicotine, in particular before ripening.