1) Don't let your focus deride you from getting a more well-rounded education. That said, it should be an
interesting, engaging, and satisfying education. Not just "because I should".
2) And, that said, keep your eye open towards to possibility(s) of turning this into a 5 (5.5, 6?) year Masters track. I guess I'm of two minds about this. My understanding from others is that, particularly in the shorter timeframe, such a curriculum might squeeze out time and opportunity for some other topics -- you might miss things you would find of significant value. But, it can be a more direct path to a more valuable degree and perhaps to a deeper educational experience in your topic of interest -- if this does indeed mesh with your interests.
So, I'll say, look into such opportunities. Whether you ultimately decide to sign up for such a path is a matter of whether it would be right for your (and what opportunities end up presenting themselves -- or, perhaps better stated, what you can shake loose, and at what cost).
P.S. A difficult, hard, challenging professor is one thing. A crap professor is another (and, believe me, they do exist -- whether their attitude is malicious, disengaged, or even ostensibly helpful). Don't waste too much time on the latter. Nor work too hard to avoid the former; I remember one math(s) professor with a somewhat fearsome reputation, but boy did you learn! :-)