If the machine has a UTF-8 encoding (like, say, every modern system), it will try to treat filenames as valid UTF-8 strings and fail to back up files which don't fulfill that assumption. The "solution" is to run the TSM software with a single-byte locale like en_US.
I've seen a number of shops that were silently missing files from backup from old systems because of this problem.
... most do neither, but rather do ${complex thing emerging from combination of implementation details of runtime and backup tool, impossible to reproduce in any other runtime, likely platform- and environment dependent; the same backup likely restores in different ways on different machines, and the same source files create different backups on different machines; creating a backup on one machine and restoring it on another does not generally result in the same files; and I have not yet mentioned what might happen if you mount the same source file system from different platforms, because results might vary a lot; also, we are only talking about paths here, not any of the other plethora of things that can and will be different between any element in OSxFSxEnv}.
Sure it can. In this case, I'd say treating the filename as a bag of bytes is the correct way to go, as that's the way the OS treats them. Translating filenames between character sets should not be part of a backup systems job.
There are valid setups where different software on the same machine might be running with different character sets for legacy reasons. In that case there is no correct way to handle the filenames as text. But treating it as a bag-of-bytes will always work consistently.
Also, the one purpose of a backup system is to back up the files on the filesystem. If it can't back up some files that the OS considers valid, it's the backup software that failed.