Naturally, without everyone doing the same, they would not notice the effect. However this does not constitute justification for not doing your part.
The only parties that take advantage of the driver are Uber and yourself; if Uber is unwilling to take up the mantle of proper pay then the responsibility falls on you. The question becomes: does the driver have a right to be payed properly? Or, in other words, if the driver is willing to work for a lower pay, do you have a responsibility to pay them more? Some would argue that it's a free market, and that if they don't like the pay, they should find a different job. If you believe in a minimum wage however, you should make sure that all those under your service recieve a reasonable wage.