It's the energy in the passenger's body that is usually the issue.

For instance, seat belts absorb the energy instead of the body flying through a window, crumple zones increase the time it takes to push the energy into a seatbelt (which decreases the force and thus the damage).

Yeah this is correct. Less often talked about but actually very important, the amount of energy required to break something[1]. Also as ductile structures deform under load they tend to redistribute the load more evenly.

[1] Hammer a piece of glass vs a piece of rubber.

Yes, that's exactly right. Ductility is measured by how far a metal deforms before breaking entirely. Ultimate strength is measured in terms of how much pressure can be applied per unit of cross sectional area. A metal that can be elongated 50% while under 1000 MPa stress absorbs lots of energy. If a metal is less ductile, it will break after little deformation and therefore stop absorbing energy.

Note that the metals under discussion here have very high strength (~1000-2000 MPa) and relatively little ductility (~10%), but are more ductile than other materials with this strength. On the other hand there are very ductile alloys that can tolerate 50% elongation and have strengths around 1000 MPa. These are achieved by adding basically a shitload (20% or more by mass) of manganese.