If you do not need the graphical stack, there is room for optimization both in kernel and user space. On a similar board (4 ARM cores) with Ubuntu I managed to start user space applications in <2 seconds (from power on) and the complete boot (including Xorg/MATE) in <10 seconds. If you are interested, on Medium I have some write-up (same username, last post).