I'd say go through Knuth's The Art of Computer Programming in your spare time, it's got one of the widest coverages of that sort of stuff, and it's actually a lot more approachable and better written than a lot of the other similar textbooks out there. Assume it'll take you a while though.
I don't have any programming friends and so I also sometimes find it hard to keep up with a conversation that is using a large number of technical terms, as I have the mental overhead of converting those to ideas...it's not as natural as it could be with more practice.
I haven't read TAOCP but I have gone through a bit of SICP and I can recommend that. For slightly more practicality, I was doing it in Clojure, since that is more likely to be useful in the future than Scheme--for me, at least. The video course is on Youtube. I'd really like to get back at it and finish going through the entirety of it. Someday.
http://www.afronski.pl/sicp-in-clojure/2015/06/04/sicp-in-cl...
Since I have a good amount of actual development work, I'm doing Algorithms and Data Structures. Then I'll have a go at Nand to Tetris, since I have the book.
There's infinite room to grow from there. Copy the Trello board and it's a fun little game to get through. It's been fun and not as intimidating as you may think.
Good luck!
Top notch quality free course