There are many parts to the book, but in one the author goes over formal systems. It was the first time symbol manipulation had been introduced to me as a way of formalizing things like arithmetic.

By that point I was already playing with numbers and looking for patterns in the Fibonacci triangle and various series, so being able to play with just symbols was natural and fun, especially if the system is constructed in just the right way so as to have meaning that transcends itself.

Later the book meanders through lots of other wonderful topics, but the theme seems to be simple systems which add up to something more than the sum of their parts. There's something about realizing how much depth lies in a handful of simple rules that really clicks and makes you value those rules in some way. After he's primed you with all kinda of these things, he introduces lisp.

It took another step or two after GEB for the beauty of lisp to really hit me, but GEB laid the groundwork by fetishizing recursion and simple foundations, among other things.