Keywords: Optimal Control, Bellman. I'm serious.

For a first exercise, forget Dijkstra and just solve a maze by doing Value Iteration, and plot the cost-to-go at each step.

Then consider that this function doesn't have to take a graph vertex or grid cell, but could instead be some continuous function on R^n.

The next step usually is to learn about the Linear Quadratic Regulator problem, where the cost-to-go is a quadratic, and you get to do an iteration of "Value Iteration" by updating the quadratic coefficients.

To connect to physics, see how you'd write the Action Integral in these terms.