Not if you don't tell anyone about it. I think the bigger worry would be if it was declared abandoned property due to inactivity, but if you are making occasional withdrawals that might be OK.

You're right. This should be in my will.