You're missing that "& per se &" would still have been called that even if it wasn't the last item in the list. If the order was "...y, & and z" you'd pronounce it "why and per se and and zed".
Maybe it’s you who is missing what’s going on.
https://www.quora.com/What-is-the-origin-of-the-ampersand
& was named ampersand in the context of rattling off the letters of the alphabet in English: “double-you ex wye izzard/zed, and”.
“… And what?”
“And, on its own, and” (and, per se, and).
With “and” named in English, because this was an English rattling off of the alphabet.
Yes, specificially this article: http://haggardhawksblog.blogspot.com/2015/03/ampersand.html