If you train for health instead of body building, going from sedentary to a regular body weight routine is a colossal improvement.
These concerns are especially relevant for the elderly.
Getting started with a body weight routine or simple bands or kettle bell setup at home has huge advantages.
My recommendation would be to rent a rowing machine and try that first.
Edit: I should also say that I use weights like dumbbells and kettlebells at home. Not needed to start out, but very helpful.