Exactly, what is mocked as Yak Shaving is the important stuff. But try telling your non technical manager why you need to add that logging system, which requires its own testing and deployment.
In this scenario you are the one focused, the non-technical people that shuffle tasks around playing 'agile' are the handicap. It could take a couple of hours or a couple of days to get that logging done but explaining to the non technical manager (who refuses to learn) why this needs to be done is a task in itself that cannot be underestimated.