I wish impendia had left out of his otherwise excellent explanation the "squinting" part and stuck to explaining that the zeta function is defined by the infinite series where that series converges, and defined where that series diverge using a tool called analytic continuation.
In short:
Where the series obviously converge, use the summing formula.
Where the series is mis-behaving, use analytic continuation instead of resorting to weird infinite series re-ordering tricks (which I've always felt to be borderline offensive from a mathematical rigor pov).
My understanding of analytic continuation is that if a function f of the complex plane is sufficiently well behaved on a certain domain of the plane, it can be "extended" to the rest of the plane in a unique way that preserves the well-behavedness.
In the case of zeta, it can be shown that zeta obeys a functional equation that allows it to be extended everywhere.
A better explanation than mine is here:
https://math.stackexchange.com/questions/437883/what-is-the-...