Much like GPT fine tuned to not say certain things so are our brains by the parenting we receive and reactions we experience in early life.
That it we can't yet do much better than very broad brushstrokes — and that even this is a massive improvement on the hand-grenade-in-a-paint-tin that is the Scunthorpe problem — is basically why the open letter about pausing capability research for 6 months so that alignment research can at least get started in some meaningful way.
It is a technical term. It means that you train the network on one task a lot, and then you train it on a different one a little.
Often the network learns the second task faster and better than it would do if trained from a random initialisation. This second training step is called "fine tuning".
In terms of an LLM the first training might be "here are many text samples (the whole internet), predict what is the next word". While the fine-tuning training might be on a much smaller curated corpus where you show it examples where the network responds to instructions. But you could do a lot of other things. You could create a fine-tuning dataset which makes the network betitle and antagonise the user, or one which ignores the prompt and all it does is praises god, or rhymes about butterflies.
In this technical sense "fine tuning" doesn't say the resulting network is of good quality, or suitable for some particular application. Just that it was trained in this particular two step fashion.