This is an unhelpfully cynical take. The job title has "engineer" in it, so a more charitable interpretation is more serious than "AI monkey".

Using LLMs to solve real problems is not easy. Making sure that you don't introduce regressions while making improvements is difficult, and requires building and evaluating a dataset, and the necessary pipelines. It may also include diversification of LLM providers, and creating the necessary abstractions. A fundamental understanding of how LLMs work, ability to compare different architectural approaches, along with typical data engineering and software development skills would be required.

What if you want to use the LLM for Question/Answer systems that requires working with embeddings? What if you want to find a way to process data locally without sending sensitive data to the LLM provider?

This requires real engineering skills.