Tango primarily supports PyTorch, but unlike AllenNLP, is flexible enough to support other deep learning libraries as well. For example, we're adding support for JAX so we can easily leverage TPUs.
That said, you could execute sklearn. If that's what your experiment needs, it's the right thing to do. This is why it gives us the flexibility to also support Jax: https://github.com/allenai/tango/pull/313
The DL-specific stuff is in the components we supply. Like the trainer, dataset handling stuff, file formats, and increasingly, https://github.com/allenai/catwalk.
And yeah, Tango is a lot like a build script. In fact, I used to manage my experiments with Makefiles. Tango is better though. Results don't have to be single files, and they don't have to live in one filesystem either, so I can run the GPU-heavy parts of my experiments on one machine, and the CPU-heavy parts on another. The way you version your code is better than what Makefiles can do. You have actual control beyond file modification time. And of course, there is the whole Python integration stuff.