4 is being actively avoided. And combines with other numbers, like 14(to die), 174(going to die) make it a very ominous number indeed.
As to 54, maybe you are referring to 勿死(wu4 si3)? Its meaning is closer to, please don't die, than not dying, and though I am not sure, but 勿 uses to negating another word seems kinda ancient and outdated.