This is literally the point of robots.txt. It was created to allow site owners to configure how and which parts of their website can be scraped by what bot, and all the "decent" ones (Google, Bing) respect it.
Spoiler: They don't.
Google, Bing do
Was that not possible before?
no-index is a thing.
no-index is per individual page, not for an entire domain, IIUC?
You can pretty easily send no-index for an entire site by configuring it as a site-wide HTTP header.