That depends entirely on what they are listening with. One sub set of ham radio is called fox hunting which is a gameified form of radio direction finding. Some guys are really good at it. If you annoy one of them and they are persistent they can potentially track you down. The Feds of course have very sophisticated tools far and above what's available to you but if you've drawn that kind of attention you are already in deep trouble and looking at jail time. Powerful software defined radios like the RTL-SDR are inexpensive and with a PC can be used to scan broad swathes of spectrum and even decode and store transmissions. People can setup their own DIY listening posts this way. For someone with the right setup and looking at the right time they would notice you are using an encrypted transmission. To figure out where you are would involve repeated detections from multiple points. An adjacent topic is pirate radio stations. The Youtube channel Ringway Manchester has a series of videos about historical UK pirate stations and their stories. You might find it interesting.