Ah my bad, looks like it only happens when I kill the app and not when I exit the app properly. I do not know if android apps are supposed to clean up persistent notifications when they are killed or even if they can.