The encoder is a neural network which is a nonlinear and highly expressive function family. The encoder function is optimized to produce a set of "factors" which are designed to be linearly comparable. If the "factors" aren't expressive enough, you can add more (increase the embedding dimension) or make them more sophisticated (increase the encoder network complexity). There's nothing stopping you from allowing them to interact in a more complex way as well, except that you give up the ability to easily do operations like indexing.