My personal experience is that I got bored with frontend and pushing pixels, and when I tried to switch into infra work I found my incomplete picture of CS made it hard for me to grasp concepts other seem to take for granted. Maybe you don't have to have a CS degree, but even having a survey knowledge of major CS topics will at least give you more confidence.