>A pretrained model without deliberate alignment is by default aligned to the average person in the developed world, since that's what's inside the pretraining corpus - stuff on the Internet made by humans.

I don't think this is true or a useful way of thinking about it. If the training process makes the model aligned to it's content, then the models are 1. aligned to a random subset of Internet content, weighted by text volume and being easy to scrape, 2. aligned to the training process that makes models chatbots that answer your question instead of just continuing your passage in a similar style. Neither of these are necessarily good enough, IMO.

And that's taken it as a given that the training process can be said to align the models to the authors of the content by default, regardless of what that content actually is. I don't think that should actually be a given.

>You may not personally find the default alignment aesthetically pleasing, but it's humanity. We should set the initial conditions of this new era faithfully, and let it unfold naturally.

Strongly disagree, I think the assumption that natural = good is incorrect and harmful. Polio is natural. And to even call the model's "unaligned" state "natural" seems like an enormous stretch.