I think there's a fair amount of ARM that's different about each implementation.

eg: early boot - initialization of hardware is timing/order dependent

ARM SoCs are complex and often undocumented to anyone outside the company making them. The arch is fine, but many drivers are needed e.g. for clocks (and a lot of reversing has to be done). Having said that, downstream (Android) kernel will probably be fine for a simple use case such as this