The momentum you inherit from Earth isn't in the direction of the sun, but roughly perpendicular to that direction.

It's similar to a car going down the freeway at high speed that needs to make an immediate 90 degree right turn. It can't do that without slowing down first, if it did, it would slide sideways off the sideroad. It's harder in space because the car is going 30 km/s and there's no friction that slows you down.

Imagine sitting on a huge carousel spinning fast, like 100 times the speed of sound fast. Now imagine jumping off and trying to get to the center of it.

Yes, gravity is holding us back from yeeting of into space but to get closer to the sun you need to slow down. A lot. It takes way less rocket fuel to speed up enough to leave the solar system than to get to the sun.

That's not how orbital mechanics works. Objects move in elliptical orbits around the sun, not in straight lines.

https://space.stackexchange.com/a/45619

The exact same reason why planets don't immediately fall into the sun