Testing is almost unquestionably a benefit (at least to a point) and is something that can be introduced with minimum initial investments and right away gives big benefits. I'd start with that for low effort, big impact.
Then CI (but not CD) is like that too. Just have the test executed on every commit/PR to gain some confidence.
CD requires a whole culture around it, and not everything fits into this culture.
There's really nothing wrong with deploying stuff via SSH. It's not most fashionable, but if done via script and works, it's OK. No need to fight it.
Docker can be great, but people often shove it where it is not needed. E.g. if I was deploying Java (standardized VM) or Go (static bin), I might not bother. It really was invented for Python, where it is impossible to get the Python app running on anything but the developer's computer.
k8s ... mixed bag. Hefty price in complexity, for some benefits that are sometimes important, sometimes not.