Matter simplifies this. It defines the API layer. You can use Thread without Matter, at which point you basically have Zigbee + IPv6, but the power comes with Matter since now every device is speaking the same language and can actually understand each other.
Yes you can, I did that with Ikea, Philips and Innr brands. No hub, not even Z2M involved. Yes, as you say they do need to agree on a "protocol" and AFAIK they are all following Philips lead on that, but they can totally work in a P2P fashion without any hub. They negotiate their own key, you just need to pair them with a very close distance (less than 5cm approx).
Technically Zigbee _also_ defines an API layer -- the Zigbee Cluster Library, or ZCL -- but that's more like an opt-in standard you _could_ implement, rather than any hard requirement. And no surprise, the Matter Cluster Library Specification, being authored by the same CSA that made ZCL, is eerily similar to ZCL...
But as I understand it, you're right that Matter is essentially "hey everyone, let's _actually_ standardize around a common application layer". It isn't technologically revolutionary (the building blocks have been around for more than a decade), but it's a better packaging of it all.
Source: My employer has been involved with Zigbee and other low-power network technologies for a long time.
That works, I am doing the same. But the average consumers don't want to be bothered to run HA, they want things to work out of the box with minimal fuss setting up or operating. This usually meant having the Philips hub, the IKEA hub, the Samsung hub, etc.
> That's enough a proof that the issue was not in the protocol
For sure not in the Zigbee protocol, which is standard. The differences are in the logical communication protocol, at application level. Each manufacturer wanted to fully control their product, with no alignment with other manufacturers, which made devices and hubs mostly incompatible outside of each ecosystem. This is what Matter is looking to fix. One controller coordinating over a standard protocol a bunch of IPv6 devices connected via WiFi, Ethernet, or Thread.
And best part, Matter certification means the devices have to be able to operate locally. No more "cloud polling" [0] type integrations even for basic functions.
[0] https://www.home-assistant.io/blog/2016/02/12/classifying-th...