It seems like only clicking the small text link inside it correctly sets the cookie to remember that the user wanted it gone.
This is particularly annoying because it is binded to the scroll so it lets you scroll a few line before displaying again.
What browser are you using? What adblockers? Any special browser settings?
Edit: I see a potential cause of issues in the scroll handler, and I just deployed the change, so perhaps it's fixed on a refresh?
Chrome: Version 73.0.3683.86 (Official Build) (64-bit)
There's at least someone else with the same behavior: https://news.ycombinator.com/item?id=19643824
I misunderstood your text description and now I know exactly what you mean. It never occurred to me that people click outside the modal rather than on the "x". In retrospect it seems obvious that they would. Will fix.