This is a great (pre-GPT-4) post that goes into the system prompt and its effects: https://minimaxir.com/2023/03/new-chatgpt-overlord/

Are you using any system prompt in your API calls? Note that it does have an explicit domain separation from the user and assistant roles.

I suspect it plays a greater role in GPT-4 as well, i.e. was trained on a wider variety of system prompts to guide the output of the model.