I think e needs to be coprime with phi(N) rather than N itself. This is so you can find d = e^-1 mod phi(N), which would otherwise not exist.

Of course, if e shares a factor with N, you have bigger problems.