That sounds like you're visualizing the lever being on your side of a pivot point, so when you push it down the other side goes up. Feels natural enough to me!
Usually its just Y-inverted for me though.
This is a third-person camera. First-person is strictly seeing from the character's eyes.
I was on a pretty steep hill also, so sometimes forward literally was down.