This is not a book, but we did write up some documentation for developers that gives a nice overview of how the optimizer works:

https://github.com/cockroachdb/cockroach/blob/master/pkg/sql...

Might be worth a read if you're interested in implementation techniques.