Derivative at a Discontinuity
alok.github.io
alok.github.io
The thing that piqued my interest was the side remark that the Dirac delta is a “distribution“, and that this is an unfortunate name clash with the same concept in probability (measure theory).
My training (in EE) used both Dirac delta “functions” (in signal processing) and distributions in the sense of measure theory (in estimation theory). Really two separate forks of coursework.
I had always thought that the use of delta functions in convolution integrals (signal processing) was ultimately justified by measure theory — the same machinery as I learned (with some effort) when I took measure theoretic probability.
But, as flagged by the OP, that is not the case! Mind blown.
Some of this is the result of the way these concepts are taught. There is some hand waving both in signal processing, and in estimation theory, when these difficult functions and integrals come up.
I’m not aware of signal processing courses (probably graduate level) in which convolution against delta “functions” uses the distribution concept. There are indeed words to the effect of either,
- Dirac delta is not a function, but think of it as a limit of increasingly-concentrated Gaussians;
- use of Dirac delta is ok, because we don’t need to represent it directly, only the result of an inner product against a smooth function (i.e., a convolution)
But these excuses are not rigorously justified, even at the graduate level, in my experience.
*
Separately from that, I wonder if OP has ever seen the book Radically Elementary Probability Theory, by Edward Nelson (https://web.math.princeton.edu/~nelson/books/rept.pdf). It uses nonstandard analysis to get around a lot of the (elegant) fussiness of measure theory.
The preface alone is fun to read.
Imo, the informal use is already pretty close to the formal definition. Formally, a distribution is defined purely by its inner products against certain smooth functions (usually the ones with compact support) which is what the OP alluded to when he said:
> The formal definition of a generalized function is: an element of the continuous dual space of a space of smooth functions.
That "element of the continuous dual space" is just a function that takes in a smooth function with compact support f, and returns what we take to be the inner product of f with our generalized function.
So (again, imo) "we don’t need to represent it directly, only the result of an inner product against a smooth function" isn't that distant to the formal definition.
Here are two “test functions”-
- we learned much about impulse responses, and sometimes considered responses to dipoles, etc. However, if I read the Wikipedia article correctly (it’s not great…), the theory implies that a distribution (in the technical sense) has derivatives of any order. I’m not sure I really knew that I could count on that. A rigorous treatment would have given me that assurance.
- if I understand correctly, the concept of introducing an impulse to a system that has an identity impulse response, which implies an inner product of delta with itself, is not well-defined. Again, I’m not sure if we covered that concept. (Admittedly, it’s been a long time.)
- If delta(x) is OK, why is delta^2(x) not OK?
I don't know what kind of justification you expect. There's a Dirac delta sized "hole" on linear algebra, that mathematicians need a name for. It's not like we can just leave it there, unfilled.
When we are doing signal processing the Dirac delta primarily comes about as the Fourier transform of a constant function, and if you work out the math this is roughly equivalent to a sinc function where the oscillations become infinitely fast. This distinction is important because the concentrated Gaussian limit has the function going to 0 as we move away from the origin, but the sinc function never goes to 0, it just oscillates really fast. This becomes a Dirac delta because any integral of a function multiplied by this sinc function has cancelling components from the fast oscillations.
The poor behavior of this limit (primarily numerically) is the closely related to the reasons why we have things like Gibbs phenomenon.
I once saw in an elementary calculus book a note after the proof of a theorem about differentiation that the converse of the theorem was also true but needed more advanced techniques than were covered in the book.
I checked the advanced calculus and real analysis books I had and they didn't have the proof.
I then did some searching and found mention of a book titled "Differentiation" (or something similar) and found a site that had scans for the first chapter of that book. It proved the theorem on something like page 6 and I couldn't understand it at all. Starting from the beginning I think I got through maybe a page or two before it got to my deep with my mere bachelor's degree in mathematics level of preparation.
I kind of wish I'd bought a copy of that book. I've never since been able to find it. I've found other books with the same or similar title but they weren't it.
⸻
1. Yes, this is mathematically the reason why black holes are referred to as singularities.
>which is why they’re defined via limits
They're defined via studying f(x+h) - f(x) with a limit h -> 0. But, your example is taking two limits, h->0 and x->1, simultaneously. This is not the same thing.
f'(c) = lim_{h → 0} (f(c + h) - f(c))/h
The definition would not make sense if f wasn't defined at c (note the "f(c)" in the numerator). For instance, it can't be applied to your f(x) = (x² - 1)/(x - 1) at x = 1, because f(1) is not defined.
And it's a standard result (even stated in Calc 1 classes) that if a function is differentiable at a point, then it's continuous there. For example:
5.2 Theorem. Let f be defined on [a, b]. If f is differentiable at a point x ∈ [a, b], then f is continuous at x.
(Walter Rudin, "Principles of Mathematical Analysis", 3rd edition, p. 104)
Or:
Theorem 2.1 If f is differentiable at x = a, then f is continuous at x = a.
(Robert Smith and Roland Minton, "Calculus -Early Transcendentals", 4th edition, p. 140)
It's true that your f(x) = (x² - 1)/(x - 1) has a removable discontinuity at x = 1, since if we define g(x) = f(x) for x ≠ 1 and g(1) = 2, then g is continuous. Was this what you meant?
and yes, by defining a new function with that hole explicitly filled in with a defined value to make it continuous is the typical prescription. It does not imply the derivative exists for the other function as the other post posits.
That's right. In the example f(x) = (x² - 1)/(x - 1) for x ≠ 1, if we further define f(1) = 0, the function is now defined at x = 1, but discontinuous there.
> ... if the limit of the derivative from both sides of the discontinuity exists and is equal, the derivative exists.
(You probably mean "both sides of the point", since if there's a discontinuity there the derivative can't exist.) Your point that, if the left and right-hand limits both exist and are equal, then the derivative exists (and equals their common value) is true for all limits.
Also, there's a difference between the use of the word "continuous" in calc courses and in topology. In calc courses where functions tend to take real numbers to real numbers, a function may be said to be "not continuous" at a point where it isn't defined. So f(x) = 1/(x - 2) is "not continuous at 2". But in topology, you only consider continuity for points in the domain of the function. So since the (natural) domain of f(x) = 1/(x - 2) is x ≠ 2, the function is continuous everywhere (that it's defined).
However, you could also (probably) define the derivative as lim_{h->0} (f(c+h) - f(c-h))/2h, so without needing f(c) to be defined. But that's not standard.
Although this gives the right answer whenever f is differentiable at c, it can wrongly think that a function is differentiable when it isn't, as for the absolute-value function at c = 0.
A function is continuous at x = a if it is differentiable at x = a.
You do understand the concept, but your precision in the definitions is lacking.
It's called "removable" because it can be removed by a continuous extension - the original function itself is still formally discontinuous (of course, one would often "morally" treat these as the same function, but strictly speaking they're not). An important theorem in complex analysis is that any continuous extension at a single point is automatically a holomorphic (= complex differentiable) extension too.
-----
Concerning the Dirac delta example: I think this is probably a pleasant way of using a sequence of better and better approximations to the Dirac delta. Terry Tao has some nice blog posts where he shows that a lot of NSA can be translated into sequences, either in a high-powered way using ultrafilters, or in an elementary way using passage to convergent subsequences where necessary.
An interesting question is: What does distribution theory really accomplish? Why is it useful? I have an idea myself but I think it's an interesting question.
That can give wrong answers because derivative of the limit is not always the limit of the derivative.
When modeling phenomena with Dirac delta, I think the question becomes do I really need a discontinuity to have a useful model or can I get away with smoothening the discontinuity out.
Also when Fourier transforming over the whole real line (not just an interval where the function is periodic), one has identities that involve delta functions. E.g. \int dx e^(i * k1 * x) e^(-i * k2 * x) = 2 * pi * delta (k1 - k2).
The question is why distribution theory is a particularly good approach to notions like the Dirac delta.
The limit of the Gaussian function as variance goes to 0 is not a function, but it is a distribution, the Dirac distribution.
Some distributions appear in intermediate steps while solving differential equations, and then disappear in the final solution. This is analogous to complex numbers sometimes appearing while computing the roots of a cubic function, but not being present in the roots themselves.
In impulse/constraint mechanics, when two objects collide, their momentum changes in zero time. An impulse is an infinite force applied over zero time with finite energy transfer. You have to integrate over that to get the new velocity. This is done as a special case. It is messy for multi-body collisions, and is hard to make work with a friction model. This is why large objects in video games bounce like small ones, changing direction in zero time.
I wonder if nonstandard analysis might help.
Integration can be done with its own special arithmetic: Interval arithmetic. I base this suggestion on the fact that this is apparently the only way of automatically getting error bounds on integrals. It's cool that it works.
NSA does not work with a computable field so it's not directly useful. But at the end of the article, there's a link to some code that uses the Levi-Civita field, which is a "nice" approximation to NSA because it's computable and still real-closed. You might be able to do an "auto-limit" using it, in a kind of generalisation of automatic differentiation. This might for instance turn one numerical algorithm, like Householder QR, into another one, like Gaussian elimination, by taking an appropriate limit.
I don't know if these two things interact well in practice: Levi-Civita for algebraic limits and interval arithmetic for integrals. They might! This might suggest rather provocatively that integration is only clumsily interpreted as a limit of some function. Finally tbh, I'm not sure if this is the best solution to the friction/collision detection problem you're describing.
If you can get realistic billiards breaks, you're on the right track.
I've never learnt this level of maths formally, but it's been an interest of mine on and off. And this post explained it very well, and pretty understandably for the laymen.
> You’re probably familiar with the idea that each piece has infinitesimal width, but what about the question of ‘how MANY pieces are there?’. The answer to that is a hypernatural number. Let’s call it N again.
Is that right? I thought there was an important theorem specifying that no matter the infinitesimal width of an integral slice, the total area will be in the neighborhood of (= infinitely close to) the same real number, which is the value of the integral. That's why we don't have to specify the value of dx when integrating over dx... right?
The theorem tells us that the area of the extended interval that uses a hypernatural number of slices has the same real part as the area of the exact interval. It doesn't tell us that the exact interval contains a hypernatural number of slices.
Plus the ancient mathematicians did very well with just their intuition. And more to the point, I cared much more about building (hyper)number sense than some New Math “let’s learn ultrafilters before we’ve even done arithmetic”.
They did. But they also got things wrong, such as thinking that pointwise limits are enough to carry over continuity (see here for this and other examples: https://mathoverflow.net/a/35558). Anyway, mathematics has changed as a discipline, we now have strong axiomatic foundations and they mean that we can, in principle, always verify whether a proof is correct.
As for whether that's worth it or not, I have no strong opinion.
Or put differently - here you can kinda ignore the deeper formalities and still be productive, whereas with distributions you actually need to sit down and pore over them before you can do anything.
That said, I'm curious why infinitesmals never took off in physics. This kind of quick, shut-up-and-calculate approach seems right up their alley.
I don’t know, this feels like a math “hold my beer” moment. Math is infinitely deep and interconnected, but you have to start somewhere, on solid ground.
I was not being facetious above - the issues that i mentioned above are actual problems when you make calculations. But let’s ignore those issues for a second.
So you found the “derivative” of a single, arbitrary chosen representative of an infinite family of functions. What if you chose (tanh(Nx)+1)/2? What if you chose Logistic(N^2 x) instead of Logistic(N x)? You’d get different derivatives. In fact any function (up to additive constant) whose integral of the neighborhood of 0 is 1 would work there. What use are the values you are calculating if they reflect your choice and not anything inherent to the problem?
As for distributions, i picked up and read a small 100 page penguin “leaflet” from the library during my undergrad that went through the subject rigorously (and with plenty of examples). It’s not that different from working rigorously with probability or real analysis. And at the end, in applications we indeed are usually interested in integrals, not derivatives which we have not even defined. At the end of the day, you have a [X=weak L^infinity(R)] function (heavyside). You look at the dual space and since we established don't really need the deep theory, believe me when i tell you that the correct space is the space of test function on R (X’=infinitely smooth, compact support, bounded integral). Each of those conditions is simple for our simple example of R. The inner product is via integral.
Formally speaking elements of X are equivalence classes of sequences of functions and are not really defined pointwise, but neither was the NSA example. There we had to choose an arbitrary representative hyperreal function and here we may identify pointwise defined functions with the classes of the constant sequences of those functions.
using integration by parts it is simple to show that <F,G’> = <F’,G> if F is continuously differentiable on G’s support. Let us formally define in this way the weak derivative for functions that are not traditionally differentiable, if such an element exists an is unique that satisfies all the integral relations. However note that differentiation is an linear isomorphism on the space of test functions and so weak derivative indeed exists and is unique. Furthermore
We can also define elements of X poinwise by identifying F(x) with the limit <Txn,F> as n grows if it exists and is independent of the sequence Txn where Txn is a sequence of functions with support tending to {x} and constant integral 1. It is a simple exercise to show that for “normal” functions this holds, and by above we can poinwise define derivatives this way as well.
What about our H(x)? it is an exercise to check that by pointwise we get what we should outside of 0. What about the derivative at 0? Well, do the exercise above with <T0n’,H> and we see that it is penrose undefined. Decidedly not even necessarily infinite, just undefined. However, integration by parts shows that <T,DH>=T(0) ie dirac delta at 0.
Aside from all the theory that i kinda gave handwavingly much like OP in the post, the mechanics are simple integration by parts to get the only stuff that’s “real” here, which are the integrals. in NSA we haven’t even defined those. How will knowing what infinity i will get at 0 given an arbitrarily chosen representative for H help me?
Do your results depend on ZFC? stronger axioms? At what level of infinity do we stop? You can brush aside the formalities but then what better is this approach than physicists?
They all differ pointwise by an infinitesmal! This flexibility is a feature, not a bug.
Of course, I agree with you that mathematical progress should proceed on rigorous grounds, and there is a lot to be proven here. But my point is mainly that this is so easy you can just go and see what happens in these cases yourself without much trouble. For applications you really don't have to care.
I've done a course in analysis that covered distributions, but your reply made me chuckle. You've told me that distributions are just as simple and then proceeded to dump paragraphs of jargon at me. L^infinity? Dual space? Support? Penrose defined? Inner product via integrals?
(to be clear, I know what you're talking about, but our hypothetical high school student will have a lot more luck moving infinitesmals around, I guarantee it)
> Do your results depend on ZFC? stronger axioms? At what level of infinity do we stop? You can brush aside the formalities but then what better is this approach than physicists?
For physics, just pick a representative, compute and do some sanity checks. Who is ZFC? =P
Just kidding. Anyway, as you probably know the deeper theory of nonstandard analysis has been worked out in detail already, so if you want to get stuck in the weeds there are answers out there.
No, they do not differ by an infinitesimal. You picked an arbitrary infinite N and found the derivative to be N/4. What if you picked N^2? or 2^N? or some upper limit set whose existence is stronger than choice? You get a different derivative every time and they all differ by an infinity between them. Good luck explaining that to high school students.
Moreover, working with equivalence relations is never a feature of any theory. Having to prove independence from representative at every step is not a feature, as you clearly demonstrate by making the mistake above.
> I've done a course in analysis that covered distributions, but your reply made me chuckle. You've told me that distributions are just as simple and then proceeded to dump paragraphs of jargon at me. L^infinity? Dual space? Support? Penrose defined? Inner product via integrals?
All concepts that are simple to define and understand. Majority of physicists likely understand well. Those that don’t, could.
Paragraphs of jargon? i’ve rigorously proven and justified my further assertions, at a similar level to OP and above what i’ve seen in some physics lectures.
I deliberately decided to avoid defining the above “jargon” terms after considering doing that to avoid extending the already long comment. I decided this because they are simple and a curious mind could quickly understand them by browsing wikipedia.
You’re welcome to ignore them and to go and compute derivatives and integrals just as mechanistically as in NSA (and I repeat, we haven’t even mentioned integrals in NSA. Good luck defining what are measurable functions on the hyperreals to your hypothetical AP high school students).
And to boot we never have to deal with any quantities that are not real measurable numbers. Anything we care about we can compute, more easily (integrals? integrals??) this way.
That is not to say that this isn’t an interesting theory that should be studied - just that it is quite the opposite of a “simplified” approach to general functions
> What if you picked N^2? or 2^N? or some upper limit set whose existence is stronger than choice?
I'm not sure what you mean about picking N to be an upper limit set. N is a hyperreal here, not an ordinal. There aren't really set theoretic difficulties, you can easily construct a model of the hyperreals in ZFC.
It doesn't matter what representative you pick for your Heaviside function - so long as it differs pointwise from the standard Heaviside function by infinitesmals you will get a delta function by differentiating it in NSA. And continuing to differentiate it will give you the higher multiple moments. This is what I meant in my previous response.
It's useful to have the choice because depending on what you want to model you can have non-standard functions that "go to infinity" twice as fast as other functions, for instance. Taking the equivalence class destroys that information, which is sometimes useful, and sometimes not. If all you care about is a small computation you can just pick a representative and move on, I don't think it's a big deal.
Anyway, I'm going to leave this discussion now - we're not really bickering over anything important to my mind. Use the tool you like! Personally I'm having fun playing with NSA right now. Thanks for your time.
The specific infinite results you get depend on the representative you choose and if you want to calculate integrals which you have yet to define, you need to work harder. On the other hand, if you care about modeling specific infinities for applications you are very likely also someone who needs and wants the actual theory.
As someone with mathematical background indeed i repeat that i think this theory is interesting and worth studying. Just not that it simplifies anything.
Integration is very similar, by the way - you just do Riemann sums with a hyperfinite number of partitions of infinitesmal width. It sounds weird, and it is, but it makes it very easy to understand why integrating f(x) against a delta function gives you f(0), for instance, without having to justify it with limits or a bunch of deeper theory.
https://mathoverflow.net/questions/91646/surreal-numbers-vs-...
See Ehrlich’s answer.
There it is.