Wikipedia has a light page on the topic.
https://en.m.wikipedia.org/wiki/Buddhism_and_violence
Not mentioned in the above link is the Tibetan serfdom controversy.
https://en.m.wikipedia.org/wiki/Serfdom_in_Tibet_controversy
Regarding you second point, I am not sure I see the value of saying "Buddhism is not a religion if you ignore the religious bits"... you might well say that about any religion. There are plenty of secular Muslims and secular Jews for example, but that does not make Islam or Judaism any less of a religion.