At least for ^ I was always told that the ADM3a terminal keyboard had a HOME key with ~ and ^ on it, which is why ~ means "home directory", and ^ means "home of the line".
https://www.hillelwayne.com/post/always-more-history/
Whether that explains the use of tilde and caret, I dont know.