A docker container is instantiated from a docker image, it's similar to object oriented programming: an object is instantiated from a class. Want to manipulate containers? Use 'docker container ...'. Want to manipulate docker images? Use 'docker image ...'.
There's also a differentiation from production vs. development environments. How do you onboard new developers? If you use docker, you can put a docker-compose.yml file in the repo and when you onboard a new developer, it's as simple as 'git clone ${the_repo} && docker-compose up'. If you want to use docker in production, you're probably using Kubernetes, and I would agree that there's an increased cost in complexity. Want to scale the docker containers across multiple servers? The logical choice is to use Kubernetes to orchestrate the containers. If you're scaling vertically, sure, use your current method. Golang is fast.
As your company grows & scales, I think you'll run into deployment issues using scp for deployments. What happens if two developers deploy at the same time? Ideally you put every deployment through a single "pipe" where if two people deploy at the same time, one deployment/testing process would run after the other in a serial manner, causing the second deployment to fail if there were a conflict.
Arguing about tech turns into a religious debate at a certain point. Use docker/Kubernetes if you need to, and if you can get by without using them, don't use them. Docker is awesome for solving versioning problems and onboarding new employees. Kubernetes is awesome at scaling and deployments. If your employees all agree to use the same version of software, there's no need to use docker or Kubernetes.
But hey, at least docker/Kubernetes give you the choice to think freely and if you see some cool library that you want to use written in some obscure language or version, it's easy peasy.