I used fastai quite a bit with PyTorch and ended up feeling the same way. Great for spinning up an effective model in a few lines, but really tough to make it do exactly what you want it to do.