Agreed, using neuron layers as vectors seems like a waste. Moving to a graph-like compute model seems like it'd be more efficient.