I think the fundamental source of confusion might be due to not understanding the concept of a random variable. In his example, X is a random variable; the expectation E[X] is an functional applied to its probability mass function. Given this, it should not be surprising if it seems to behave differently to "any other function, in math".

If you understand this, I think the notation is natural:

We have a random variable X, which takes value x_i with probability p(x_i). Thus, the random variable X^2 will take value (x_i)^2 with probability p(x_i).

Given that the expection for the random variable X with p.m.f p(x_i) is defined as E[X] = \sum x_i p(x_i), it should be clear that to obtain the expectation of any random variable we must sum over the product of (value) and (probability of that value). It should also be clear that this gives E[X^2] = \sum (x_i)^2 p(x_i)

I'm confused by his comment that: p(xi) isn't, because it doesn't make any sense in the first place. It should really be just PXi or something, because it's a discrete value, not a function!

The probability mass function is a function: for a given value, it gives the probability that the the discrete random variable takes that value. To calculate the expectation we use the values obtained by evaluating the function at discrete points, but what else could we do?