Because that's how "dirname(3)" is implemented in glibc, except it searches '/' instead of '\'. Here all character encodings share the same code.
I'm struggling to imagine how this failure would manifest. Can you give an example of how dirname() would fail? What combination of existing file/directory name, and usage of that function, would not work as expected?
Edit: I'm also a bit confused how this counts as being a problem for "modern Linux systems" - wouldn't it have always been a problem for all Unix-based OSs?