The most interesting part of YT is Cypress. I'm particularly interested in how they make their master cluster horizontally scalable.
However, this approach proved to be non-scalable as the memory amount and throughput of the master server soon became insufficient. To address this issue, we implemented Multicell technology. With Multicell, there are multiple RSMs called secondary masters that store information about chunks of the tables and their placement. The primary master still stores information about the distributed filesystem and transactions but is now single and non-sharded.
After a few years, the masters became overloaded again, and we implemented Portals. With Portals, one can select a subtree of Cypress and place it in one of the secondary masters. This technology is used nowadays, and home directories of some active users are hosted on secondary masters.
However, we anticipate that this approach will also become insufficient in a few years. Therefore, we are currently working on a new technology called Sequoia, which stores information about the Cypress tree shape in horizontally scalable dynamic tables.
It is hard to describe all aspects of master server internals in one comment. Therefore, feel free to join our chat at t.me/ytsaurus for further discussion!
Why not just use a database for the metadata? Something that can be sharded and has transactions like YugabyteDB/Yandex-ydb/etc?
Storing Cypress nodes in the ad hoc RSMs and information about tree in k-v storage seems a good compromise that is both scalable and allowing to implement any functionality for objects efficiently.
Having said that, this definitely does look to be an impressive feat of engineering!