The proprietary layout engine is named TALA. IMO the docs are pretty transparent about TALA being closed source and behind a paywall.
TALA docs: https://d2lang.com/tour/tala/
TALA product page: https://terrastruct.com/tala/
TALA docs: https://d2lang.com/tour/tala/
TALA product page: https://terrastruct.com/tala/
It seems the 2 are distributed separately and D2 supports open-source layout engines also.