No. robots.txt is not something that is defined and enforced by the law. Just because someone came up with some 'recommendation' like robots.txt does not mean this is the law
You failed to make a meaningful counterpoint; the legal/ethical distinction was made clear in the parent post.