In the math I know, it’s not necessarily small, but it is larger than zero. https://mathworld.wolfram.com/Epsilon-DeltaDefinition.html:
“An epsilon-delta definition is a mathematical definition in which a statement on a real function of one variable f having, for example, the form "for all neighborhoods U of y₀ there is a neighborhood V of x₀ such that, whenever x in V, then f(x) in U" is rephrased as "for all ε > 0 there is δ > 0 such that, whenever 0 < |x - x₀| < δ, then |f(x) - y₀| < ε.”
(Also used in proofs. https://mathworld.wolfram.com/Epsilon-DeltaProof.html)
https://en.m.wikipedia.org/wiki/Limit_of_a_function#%7F'%22%...:
“For every real ε > 0, there exists a real δ > 0 such that for all real x, 0 < |x − p| < δ implies |f(x) − L| < ε.”
> it's something where we can handwave its qualitative size depending on what's convenient.
Aha! You’re doing physics, not math. Mathematicians only handwave when they can prove it doesn’t affect results ;-)