CNNs aren't rotation invariant on purpose. If they were, you would lose information about how features are oriented. Typically it learns features that are very sensitive to rotation, like vertical edges and horizontal edges.
My gut feeling is that the first convolutional layer's kernels, for example, would probably have a 'some are orthogonal' constraint due to this symmetry.