If you just take a magic balloon up to a certain height, once you're up there and let go fo the balloon - you just start falling back down.
You need to have a lot of sideways velocity in order to actually get into orbit, to be revolving around whatever body you're trying to get off of.
(It didn't used to make sense to me, either.)
Launching a rocket to orbit is like starting from the bottom of the well, and spinning around the slope really fast until the nickel is returned to your hand.
The gist is that gravity pulls you. To go to space you have to escape gravity. Even if you go thousands of miles above earth, gravity will still pull you. To counteract gravity, you need to go really fast sideways.
When you spin a ball attached to a string, the faster you spin, the more the ball pulls the string away from your hand. If you spin it really really fast, the string breaks eventually and the ball flies away.
The string is gravity.
But no, it's not that you would be pulled into the sun. You would simply be in a solar orbit just like the earth, unless you continued accelerating and reached solar escape velocity (~600km/s). You can think of gravity like successively larger "wells" which you must escape from if you want to go anywhere. You can't just point a spaceship at a planet and go there.
Disclaimer: You can, you just need so much power it’s like flying with a helicopter from your bedroom to the kitchen, absolutely wasteful.
http://parkersolarprobe.jhuapl.edu/The-Mission/index.php 'Journey to the Sun.'
That's for escaping from the surface of the Sun. At Earth's distance from the Sun, the velocity needed to escape from the Sun's gravity is only 42 km/s.
What the GP said is that if when you reach Earth's escape velocity, there is still a long way to go until you have Sun's escape velocity (so you are on a solar orbit), and after you reach that, there is a huge distance until Milk-Way's escape velocity, and an even huger way until you escape the Local Group.
That's not what he said. Escape velocity is higher than orbital velocity (as ythn pointed out upthread). And escape velocity from Earth is still slower than Earth's orbital velocity around the Sun, which in turn is lower than escape velocity from the Sun's gravity at Earth's distance from the Sun.
We do this all the time with probes to other parts of the solar system. Manned missions (Apollo) used a small number of earth orbits before doing the burn to get to the moon primarily to ensure that systems checked out before making the bigger step.
Unless you hit the ground, or a mountain, or a plane, or the moon, etc.