If you don't care about latency or can wait to set up a batch of inputs in one go there's an alternative method. I call it batch prompting and pretty much everything we do at work with gpt-4 uses this now. If people are interested I'll do a proper writeup on how to implement it but the general idea is very straightforward and works reliably. I also think this is a much better evaluation of context than needle in a haystack.
Example for classifying game genres from descriptions.
Default:
[Prompt][Functions][Examples][game description]
- >
{"genre": [genre], "sub-genre": [sub-genre]}
Batch Prompting:
[Prompt][Functions][Examples]<game1>[description]</game><game2>[description]</game><game3>[description]</game>...
- >
{"game1": {...}, "game2": {...}, "game3": {...}, ...}