I think the author has a mess on his hands because he's trying to do it the Python 2 way – processing text without a known encoding, which is not really possible, if you want the results to come out right.
To resolve the mess in Python 3, choose what you actually want to do:
1. Handle raw bytes without interpreting them as text – just use bytes in this case, without decoding.
2. Handle text with a known encoding – find out the encoding out-of-band from some piece of metadata, decode as early as possible, handle the text as strings.
3. Handle Unix filenames or other byte sequences that are usually strings but could contain arbitrary byte values that are invalid in the chosen encoding – use the "surrogateescape" error handler; see PEP 383: https://www.python.org/dev/peps/pep-0383/
4. Handle text with unknown encoding – not possible; try to turn this case into one of the other cases.
Also, watch Ned Batchelder's excellent talk, Pragmatic Unicode, or, How do I stop the pain?, from 2012: https://pyvideo.org/pycon-us-2012/pragmatic-unicode-or-how-d...