I don't understand why you would need custom firmware. It seems like you could go a long way just implementing back-ends for popular ML libraries in openCL / compute shaders