Hacker News Reader

I tried discussing energy drinks with it and found out that the King of Sweden has just issued an edict against Red Bull.

View on HN

That sort of thing drains any enthusiasm I have for the project. Even if it were trained exclusively on old materials, in fact, I would worry about modern footnotes and book introductions tainting the training corpus (ie: by introducing knowledge it should not have).

I do love the idea, though. It it were possible to use ideal training material, it would be close to time-travel.

As it stands, there's a limit to what one can learn from the text it produces. Much value, imo, would come from what the model does not know. Ideally it would be difficult even to have it comprehend user prompts, due to how different English is today.

I wonder if there's some benefits of reinitialising completely the final layers and then training with the new data. At least for image processing it's common to use a common backbone trained on everything, and then put a final layer or two of task-specific training on top.

Reply on news.ycombinator.com