Failing a full derivation from the ground up, a proof of the equivalence to Newtonian mechanics would be interesting.
But if your question was more about "why minimizing something yields trajectories", I personally would argue this is beyond physics. As an empirical science, physicists have seen this kind of behaviour broadly (optics, classical mechanics, quantum mechanics) and just unified it as an overarching principle.
Finally regarding the proof to newtonian mechanics, I don't have anything handy from the pure Newtonian perspective beyond the usual "minimises the lagrangian and your equations of motions look the same". However, you might be interested in proofs which show newtonian gravity as low energy approximation of general relativity. And since general relativity has a nice action formulation, it all gets nicely tied in.
Hope this helps!
My motivation for getting to the bottom of all this is to fill the gaps in my physics understanding at least up to quantum mechanics. I have a grasp of QM but I would like to have some insight into the conceptual leaps that brought us there from classical mechanics. QM works in the Hamiltonian picture and I recall from my undergrad days that you get there from a Legendre transformation on the Lagrangian (or something to that effect) so I'm trying to understand the justification of that approach before moving up the conceptual ladder.
Ideally I would like to be able to trace my way from simple postulates based on observation of the physical world all the way to QM, then maybe to QFT after that.
Physics Explained Ultraviolet Catastrophe: https://youtu.be/rCfPQLVzus4
This Veritassium Video goes back further in time and talks about Action, and the Genesis of this idea. https://youtu.be/qJZ1Ez28C-A
Chemistorian, The history of Atomic Theory. https://youtu.be/SqYPrA7upiE
That is only one approach to QM. As you rightly point out, Hamiltonian and Lagrangian approaches are always two sides of the same coin: one is only the Legendre transformation of the other and so they describe the same physics.
So to that end there is a neat QM Lagrangian representation: the path integral formulation. You can apply it to basic QM as well as to QFT or even QFT in curved spacetimes and string theory.
So if your goal is to "trace your way from simple postulates", that is a good way: assume your system can be described by an action, and go from there. It works in pretty much every scenario. In most research I've been involved, you always end up constructing generic actions whose coefficients eventually determine the behaviour of the theory.
And to get back to what I assume is your real conundrum (why do we extremize something to begin with), I just don't think there's any true answer as to why nature behaves this way.
What we can answer is: what is the action, what does it represent, what does it mean to extremize it? The short answer to this (provided by the path integral formulation I mentioned earlier) is that the action is essentially controlling a probability distribution of paths in a given geometry. In quantum mechanics, we interpret this distribution with that of actual particles. When you extremize the distribution, you essentially find the most likely trajectory, and if your distribution is peaked enough around that trajectory, then you can take this path as representative of your system when fluctuations are ignored.
So in the classical limit of QM, that trajectory is all that's left (and that would be the classical mechanics trajectory).
Interestingly a similar interpretation exists in statistical physics. If you "complexify" your time dimension, your action is again on a Euclidean (instead of lorentzian) spacetime and the time direction behaves like a circle whose radius sets a scale akin to a temperature. This might sound a bit complex but where I'm going with this is that once again, you can think of this euclidean path integral as a distribution of paths over fluctuations (this time thermal), and the extremum is this time the system's behaviour when at equilibrium.
Having read about some of the history of this idea, it seems to have been originally built on philosophical grounds, the idea that nature chooses the most harmonious path, as opposed to Newton's laws which seem to come from intuition based on observation of the world. If you keep asking "why?" in either framework you will eventually run up against an epistomological barrier which is unlikely to ever be crossed but in the case of Newton's laws, their basis in physical intuition makes them much easier (for me at least) to accept as given and take as a starting point for constructing a world model. With this being the case I think an acceptable result for me would be to find a proof of equivalence between the Newtonian and Lagrangian pictures. From my reading it seems like the derivation from D'Alambert's principle may be part of the journey.
The modern concept of 'work done' was formulated around 1850 (Eighteen-fifty). That is, we shouldn't assume that back in the days of Lagrange d'Alembert's principle was understood in the same way as it is today.
Joseph Louis Lagrange motivated his notion of potential energy in terms of d'Alembert's principle.
The recurring theme is the concept of 'work done'.
In case you hadn't noticed yet, I'm the contributor who notified you of a resource I created, with interactive diagrams.
There is this distinction: the work-energy theorem expresses physical motion, whereas d'Alembert's virtual work expresses, as the modern name indicates, virtual work.
My assessment is that using d'Alembert's virtual work is an unnecesarily elaborate approach. The same result can be arrived at in a more direct way.
There is a very interesting approach in the quantum physics book by Eisberg and Resnick, section 5.2
To arrive at the Schrödinger equation Eisberg and Resnick construct what they refer to as a plausibility argument.
The goal: to arrive at a wave equation that when solved for the Hydrogen atom will have the electron orbitals as set of solutions.
Eisberg and Resnick state 4 demands:
-1. Must be consistent with the de Broglie/Einstein postulates. wavelength=h/p, frequency=E/h
-2. Must be such that for a quantum entity followed over time the sum of potential energy and kinetic energy is a conserved quantity.
-3. Must be such that the equation is linear in \Psi(x,t): any linear combination of two solutions \Psi_1 and \Psi_2 must also be a solution of the equation. (Motivation: in experiments electron diffraction effects are observed. Interference effects can occur only if wave functions can be _added_.)
-4. In the absence of a potential gradient the equation must have as a solution a propagating sinusoidal wave of constant wavelength and frequency.
Eisberg and Resnick proceed to show that the above 4 demands narrow down the possibilities such that arriving at the Schrödinger equation is made inevitable.
To me the second demand is particularly interesting. The second demand is equivalent to demanding that the work-energy theorem holds good. The recurring theme: the work-energy theorem.
I have a (html)-transcript of the Eisberg & Resnick treatment that I can make available to you.
There is a youtube video with a presentation that is based on the Eisberg & Resnick plausibility argument.
In that video the presentation of the plausibility argument is in the first 18 minutes, the rest of the video is about application of the Schrödinger equation.
There's a PDF here: https://pages.jh.edu/rrynasi1/PhysicalPrinciples/literature/...
[1] An appetising quote:
> The author is well aware that he could have shortened his exposition considerably, had he started directly with the Lagrangian equations of motion and then proceeded to Hamilton’s theory. This procedure would have been justified had the purpose of this book been primarily to familiarize the student with a certain formalism and technique in writing down the differential equations which govern a given dynamical problem, together with certain “recipes” which he might apply in order to solve them. But this is exactly what the author did not want to do. There is a tremendous treasure of philosophical meaning behind the great theories of Euler and Lagrange, and of Hamilton and Jacobi, which is completely smothered in a purely formalistic treatment, although it cannot fail to be a source of the greatest intellectual enjoyment to every mathematically-minded person. To give the student a chance to discover for himself the hidden beauty of these theories was one of the foremost intentions of the author.
The most coherent explanation I've heard was from Feynnman [0]. As far as I understand it (and I may well not have understood it at all well), at the quantum level, all paths are taken by a particle but the contributions of the paths away from the stationary point tend to cancel each other. So, at a macroscopic level, the net effect appears to be be that the particle is following the path of least action.
> a proof of the equivalence to Newtonian mechanics
The Lagrangian method isn't really equivalent to Newton's method. Again, Feynman talks about this in [0]. It's that for a certain class of action, the Euler-Lagrange equations are equivalent to Newton's laws.
It's perfectly plausible to come up with actions that recover systems that represent Einsteinian relativity or quantum mechanics. This is the main reason (as I understand it) why it's considered a more powerful formalism.
The Hamiltonian approach in contrast is breadth-first: you single out a time axis, start with some initial state, and consider all possibilities that a particle (or field in QFT) could evolve forwards in time just a tiny bit (this is what the Hamiltonian operator does). Then you add up all these possibilities to find the next state, and so you move forwards through time by keeping track of all possible evolutions all at once. This massive superposition of everything that is possible (with corresponding amplitudes) is what you call a state (or wavefunction) and the space that it lives in is the Hilbert (or Fock) space.
So Lagrangian/path-integral: follow full trajectories, then add up all possible choices. depth-first
Hamiltonian/time-evolution: add up all choices for a tiny step in time, then simply do more steps: breadth-first
I imagine it a bit like a scanline algorithm calculating an image as it moves down the screen (Hamiltonian) vs something like a stochastic raytracer that can start with an empty image and refine it pixel by pixel by shooting more rays (Lagrangian)
This is my layman explanation anyways...hopefully it helps, even though i can't say much about their relationship in classical physics.
The universe has these weird laws, and for now, all we can do is accept them as is. But hopefully, in the future, someone will figure out deeper and simpler principles.
That is, the relations between the various formulations of classical mechanics are all bi-directional.
At the hub of it al is the work-energy theorem.
I created a resource with interactive diagrams. Move a slider to sweep out variation. The diagram shows how the kinetic energy and the potential energy respond to the variation that is applied.
Starter page: http://cleonis.nl/physics/phys256/stationary_action.php The above page features a case that allows particularly vivid demonstration. An object is launched upwards, subject to a potential that increases with the cube of the height. The initial velocity was tweaked to achieve that after two seconds the object is back to height zero. (The two seconds implementation is for alignment with two other diagrams, in which other potentials have been implemented; linear and quadratic.)
Article with mathematical treatment: http://cleonis.nl/physics/phys256/energy_position_equation.p...
To go from F=ma to Hamilton's stationary action is a two stage proces:
- Derivation of the work-energy theorem from F=ma
- Demonstration that in cases such that the work-energy theorem holds good Hamilton's stationary action holds good also.
General remarks: In the case of Hamilton's stationary action the criterion is: The true trajectory corresponds to a point in variation space such that the derivative of Hamilton's action is zero. The criterion derivative-is-zero is sufficient. Whether the derivative-is-zero point is at a mininum or a maximum of Hamilton's action is of no relevance; it plays no part in the reason why Hamilton's stationary action holds good.
The true trajectory has the property that the rate of change of kinetic energy matches the rate of change of potential energy. Hamilton's stationary action relates to that.
The power of an interactive diagram is that it can present information simultaneously. Move a slider and you see both the kinetic energy and the potential energy change in response. It's like looking at the same thing from multiple angles all at once.
One section of that will be replaced in a day or two: the last part of section 2. I completed a new diagram, that diagram will allow me to cut a lot of text. I believe the change will be a significant improvement.
Repeating the links: Page dedicated to the case of a potential proportional to the cube of the displacement: http://cleonis.nl/physics/phys256/stationary_action.php
From F=ma to Hamilton's stationary action: http://cleonis.nl/physics/phys256/energy_position_equation.p...
There are other demonstrations available that go from the newtonian formulation to Hamilton's stationary action. I believe the one in my resource is the most direct demonstration. (As in: a more direct path doesn't exist, I believe.)
(If you are interested, I can give links to the other demonstrations that I know about.)