Getting off caffeine is trivial. Adderal withdrawal can
  be very serious for many people. 

This is obviously anecdotal, but I've discontinued Adderall (by tapering my dosage to zero over the course of 3-4 days) on numerous occasions with zero withdrawal or side effects.

Quitting caffeine by the same method is quite a bit more difficult for me.