Seems possible (though very time-consuming) if you just use it to protect your hobby/toy machines. As soon as your protocol becomes important enough that it attracts the attention of human hackers and not just bots they can easily reverse-engeneer it.

By the way, I think you could view encrypted connections as a sort of automation of that practice: A crypto algorithm could be seen as a machine that generates "custom protocols" given a key...