> the social benefit is huge

It will be interesting to see if this is the case in the long run, assuming "huge" has a positive connotation in your post, of course.

If AGI comes to pass and it winds up being a net negative for humanity, then the ethics of any practice which involves freely distributing information that can be endlessly copied for very little cost must be reevaluated.