DL4J contributor here. I spoke with our team so the differences in performance are likely explained by array ordering. ND4J, for good reason, requires F ordering because of limitations in cuBLAS. While I haven't had the opportunity to closely examine the Neanderthal comparison (I'm also not a clojure user), the likely explanation is that there's an implicit ordering impacting this.
Deeplearning4j is written around F ordering and ND4J supports this. Admittedly, our ordering API is not obvious to the average user.
Here's an example test you can run yourself that demonstrates ordering: https://gist.github.com/raver119/92b615704ca1bf169aa23a6a6e7...
o.n.i.TensorFlowImportTest - Orders: CCC; Time: 11532 ns;
o.n.i.TensorFlowImportTest - Orders: CCF; Time: 2101 ns;
o.n.i.TensorFlowImportTest - Orders: CFC; Time: 10202 ns;
o.n.i.TensorFlowImportTest - Orders: CFF; Time: 1960 ns;
o.n.i.TensorFlowImportTest - Orders: FCC; Time: 10744 ns;
o.n.i.TensorFlowImportTest - Orders: FCF; Time: 1717 ns;
o.n.i.TensorFlowImportTest - Orders: FFC; Time: 10097 ns;
o.n.i.TensorFlowImportTest - Orders: FFF; Time: 1716 ns;