I hadn't got a water dispenser like that but I'd be really surprised to find that the internal reservoir and piping was made of anything except for plastic.
Some actually have coiled copper (it transmits "cold" better) - but almost all of them have a filter.
I guess I would. It seems dangerous not to for the reasons mentioned here.