I'd definitely agree that it seems valuable to have an Nx version that comes with a smaller, simpler install for development purposes. XLA is amazing, but it's not obvious that you want to have that toolchain setup for local development in all scenarios.
Would be great to see several different backends created for Nx as a way of exploring what feature sets are really needed and ensuring that the integration mechanisms are sufficiently simple and general to support multiple solutions.