That's absolutely not the case!
First of all, there's no "Linux API". The kernel does expose a stable interface but you're not going to write a game just by issuing system calls.
But the kernel is pretty much the only piece of this tech stack that exposes a stable interface. Pretty much everything else changes a lot. There are exceptions -- SDL2, for example, is great to write against. But most things aren't -- they change a lot, and they're undocumented. This tends to result in duplicate code that goes out of date and results in all sorts of strange quirks and bugs. phk has a good summary of this here: https://queue.acm.org/detail.cfm?id=2349257 .
As for drivers: there's one set of drivers that aren't baked in and they're critical for games, namely nvidia drivers. Anyone who's done graphics programming on Linux has their own horror stories about nvidia's stack. I won't bore you with my own, suffice to say that things are never, ever as easy as "you just write against the OpenGL API and you're set".
"Historically", I'm not aware of any major console that's built on top of a stripped Linux kernel. Sony has been using their own franken-BSD since at least the PS3. Don't know about Nintendo but given how much effort has gone into the occasional port for Nintendo hardware, I doubt they're using Linux.
Linux has been only recently adopted by some console devices, and those are basically embedded systems. You target a single hardware platform (with a single, known set of quirks), a stable kernel, a stable driver, a stable set of userland libs. It's not great -- backporting kernel fixes is notoriously finicky, for example. But it's a scheme that, at the very least, allows you to write code to implement whatever functionality you want to implement, instead of wasting time impedance-matching between userland APIs.