It is indeed compiled with CONFIG_KEYS=y. Does this protect me against this issue? I'm not sure what this means.
EDIT: The obvious question I should have asked is which distro you are running. Also, as others have pointed out, hoping that the attacker can't read kallsyms from the machine he's attacking is not really a good defense plan.