Just like a battery can look like a capacitor under some conditions it really isn't a capacitor.
Capacitors and batteries can be described using the same basic laws, even though the fundamental mode of operation is completely different.
They have the same basic properties:
- internal resistance
- a certain capacity
- a breakdown voltage
- you can charge them
- you can discharge them by connecting the terminals (hopefully through some suitable load)
- both exhibit self-discharge
- have a certain amount of inductance
And yet the one is a real capacitor whereas the other one is an electro-chemical device, only when you open one up or when you charge/discharge it a large number of times can you see that there is a real difference between the two, superficially they are interchangeable for some applications.
Does that get it right?