Ask HN: Does robots.txt serve a real purpose?

Robots.txt is treated like a fundamental piece of SEO, but I have never seen a straight answer on what its purpose is.

There are clearly lots of ways it can hurt SEO if used incorrectly, but is there a scenario where a site gets more organic traffic because of robots.txt than it would without it?

Often, people say it should be used to hide duplicate content. However, Google has been pretty clear that duplicate content is only penalized when it is intended to deceive. And they explicitly say not to use robots.txt to block duplicate content.

>Google does not recommend blocking crawler access to duplicate content on your website, whether with a robots.txt file or other methods.

https://support.google.com/webmasters/answer/66359?hl=en

It is also a poor way of blocking sensitive pages that you don't want to appear in the SERPs. A noindex meta tag is better.

Is there a good example where robots.txt helps SEO, or where it serves any other real purpose?