This is where you went wrong a capacitor is not a light bulb. Please take a look at this circuit: http://imgur.com/gallery/sOZ1F
The first graph shows the voltage and current across the capacitor when it is charged using a 10v source. The second shows the same parameters with a 5v source. The voltage source is cut-off by the zener diode at 4.7v in both cases.
The point I am trying to make is that the capacitor can be charged at any voltage as long as you cut off the supply when the voltage across the capacitor reaches a certain threshold. Using a higher initial voltage will only charge your capacitor faster it will not necessarily destroy it.
Note1: Voltage across capacitor != supply voltage.
Note2: The above circuit was simulated using LTSpice.