Most apps don't use it these days for the same reason most apps don't use any special features of Windows: they aren't really Windows apps at all even if they may be started by running an EXE, but rather UNIX/Mac/web apps with Windows ports. Every platform has a registry-esque equivalent, albeit often in a different form:
- macOS has the user prefs system which gets mapped to .plist files. There's no central database but an individual .plist file is somewhat similar to a registry directory, a bit more expressive actually.
- web has cookies/localStorage
- Closest equivalent on desktop Linux is gconf which was an attempt to build a registry like system but it never took off outside of GNOME.
Anyway the registry didn't fail. It was and to some extent still is a competitive advantage for Windows especially in work contexts because Microsoft has tools that let you set registry keys for your users globally, and for apps that store settings there, it means you can bulk configure apps using a uniform approach that's mostly free of merge conflicts instead of mucking around with text files. For example Chrome can be configured with registry keys to some extent and I suspect if it allowed more control via the registry, admins would use it.
As to why it exists and where it came from - classical filesystem designs tend to struggle with lots of tiny files. The registry can be seen as a kind of hack that implements a filesystem designed specifically for tiny weakly typed files. It'd be nice to have one FS that could do everything, but there's been little interest in that since Hans Reiser went to prison. Note that the registry is accessible to kernel mode code so is useful for configuring drivers, and the kernel takes care of enforcing basic structure (you don't really want to be parsing random textual formats in driver code but registry access is pretty safe).
Microsoft gives their rationale for it here:
https://learn.microsoft.com/en-us/previous-versions/windows/...