Charge more. Charge enough that you aren't seen as just another warm body assembly line worker. Keep charging more until the attitude towards you changes.
If you can, find out how much the middle-man company is farming you out for. Let's say you once did a contract through ManPower for ABC company. ABC company needs some labor, and you did a good job, so they reach out to you directly. You should be billing what ManPower billed your time for, not what you billed ManPower.
This is generally only possible to do through smaller businesses. Big corporations typically have contracts where they require you to go through one of their preferred vendors.