Statistics is taking a larger and larger part in many math curricula and is quite aided by the use of a graphing calculator. The AP Stats coursework and exam also assumes you will have one.
I happen to be a physicist too and while I’m not an experimentalist, I’ve been through plenty of experimental training, and have participated in real world data analysis projects. Never once have I seen any physicist doing any statistics with a graphing calculator (I did see a few when I taught undergrads mostly from other departments, so there’s that).
At the end of the day, if you want to remove the calculator from the statistics classroom you probably also have to remove the standardized test.
It does cost more to mark those tests than pure multiple choice though.
I am honestly shocked that there are any operators that aren't Extra class.