meshoptimizer [1] is an OSS implementation of meshlet generation, which is what most people think of when they think of "Nanite's algorithm". Bevy, mentioned in a sibling reply, uses meshoptimizer as the generation tool.
(Strictly speaking, "Nanite" is a brand name that encompasses a large collection of techniques, including meshlets, software rasterization, streaming geometry, etc. For clarity, when discussing these concepts outside of the context of the Unreal Engine specifically, I prefer to refer to individual techniques instead of the "Nanite" brand. They're really separate, even though they complement one another. For example, software rasterization can be profitably used without meshlets if your triangles are really small. Streaming geometry can be useful even if you aren't using meshlets. And so on.)