I had one of those $50 task chairs when I started my business. After nine months I had incredible lower back pain. It felt like I was seventy years old.
I'm the cheapest bastard in the world, so I looked long and hard to find a replacement that wasn't that much more expensive, but eventually I ended up biting the bullet and buying a Steelcase Leap [1]. It was expensive -- $900. But looking back on the decision a year and a half later, it was totally worth it. The back pain went away nearly overnight.
In other words: on this, don't cheap out. If you're looking for a chair that's going to be your primary work chair, you're going to be spending a significant chunk of your life sitting in it. It's worth spending the money to get something good. It doesn't have to be a Leap, there are lots of options, but resist the temptation to be penny wise and pound foolish.
[1] http://www.steelcase.com/en/products/category/seating/task/l...