>but to me by far the biggest problems have to do with the entire architecture

data engineers are dependent on software engineers. and the software engineering is the more difficult part IME.