So the best and the brightest did not tend to go into software and the typical state of software engineering in a lot of big japanese companies was extremely bad... It's getting better but it's still not great.
Nintendo also has a tendency to outsource a lot of their software development (IIRC the SDK for the wii and the DS was outsourced to intelligent systems who themselves then outsourced part of it)
This is of course a generalization as things goes and there are great things coming out of Japan (hello Ruby!) but having lived there, there was a marked difference between the software engineering culture in Japan and the one I saw in the US and Europe, with Japan being easily a decade or more behind.
On top of that, programming is done in English, which just adds cognitive burden. You could make a Japanese programming language, but those haven't gone anywhere, I think in part because you'd need to convert the characters to kanji as you type and that would take longer than just typing in Roman letters.
The “encoding problem” for Asian languages is real, but it goes back earlier than that, as it emerged with typewriters. See Thomas Mullaney’s “A history of the Chinese typewriter”, and Nanette Gottlieb’s “Word Processing Technology in Japan” for more. By the late 90s/00s it wasn’t a significant primary factor (that is if it was a factor, it was due to cultural inertia, and not lack of access to technology itself).
I think the problem here is looked at backwards. It’s not that Japan sucks at software - it’s that the US/North America got extremely good at software in the last 20 years (by a combination of factors that feed into each other, as always: the best research labs are in the US, the tools are built in the US, the companies that make use of them for competitive advantage are in the US, the economic context most favorable to founding such companies is in the US, etc). The rest of the world is catching up, but the gap is still visible.
But anyway, the problems of software and online-service in japan are old, very old. To some degree it originates in their late adaption of PCs with broader parts of society, to some other degree their early adaption of online-services in the age of featurephones including all the horrible patterns.
Japans society developed quite different in those areas, and you see it all the time with their software, culture and decisions. Nintendo is not really special there.
Here's a good wiki with a ton of information on the kernel/os: https://switchbrew.org/wiki/Main_Page
EDIT: Here's some AOSP usage: https://switchbrew.org/wiki/Nvnflinger_services https://switchbrew.org/wiki/Display_services and apparently some AAC codec code.
Meanwhile Nintendo still hasn't caught up with Xbox Live from the original Xbox.
I can't help but think that this is intentional from Nintendo. There are many many online and local wireless features on the Wii/ and 3/DS that are no longer present on the Switch. They seems to start over with every platform. I get the impression though that Nintendo doesn't want communication features that are not "safe" for younger players. It is really frustrating though, especially with stay-at-home, that my kids can't communicate with friends on their gaming platform. The Wii U was a step in the right direction and they scrapped that.