It does hijacks shell history. If I press the up arrow, I expect _my shell_ to handle it how I configured it, not your tool. Because you’re saying your tool is a shell prompt!
A shell prompt is the program that runs before and after every shell command to show me contextual information like the current directory, and the exit code of the last command.
If you want this to be include customizations beyond the prompt, stop calling it a prompt.