Hacker News Reader

Comment by gambler | Hacker News Reader

I'm as so tired of trying to read these "deep learning" AI papers that deliberately obfuscate what they did and didn't do. Often by using deliberately ambiguous terminology, over-explaining the domain and immediately flooding you with low-level detail even in high-level descriptions.

Each paper should start with unambiguous description of:

1. What are the inputs of the model.

2. What are the outputs of the model.

3. What is the overall size of the model. Size, not parameter count.

4. What part of the domain has been manually encoded into the architecture and what has been learned over the training period.

5. What are the restrictions on the domain compared to real life.

6. How the performance is evaluated.

This should be on the first few pages. I.e. the descriptions of what the model does should precede the description of how it does it.

View on HN

Also, the following paragraph is very misleading:

> OpenAI Five won 99.4% of over 7000 games.

The players in those remaining percentages played repeated rounds against the AI and eventually started winning more often than not. The AI had only one strategy (deathball) and once top-skill-tier players learned how to play against it, they had a >50% chance of winning.

That's not misleading if it's literally true.

Something being literally true but causing people to think something that isn’t true is pretty much the very definition of misleading.

You don’t say something is misleading if it isn’t true. You say it is untrue.

It's misleading because many people will read that and assume that the AI is nigh-unbeatable by players. But that's only the case against players who haven't played against it before.

FWIW I read it as just meaning that the AI is very impressive, but might or might not be competitive at the highest levels of play.

Like, if you took a world-class team and had them play against random opponents I'd be surprised if they lost more than one game in a hundred.

In comparison to chess, according to the ELO guidelines where a beginner has an ELO of 800, an average player is ~1500, a professional is ~2200, and only four people have a rating of ~2800, then we'd expect Magnus Carlsen to almost never lose against an average player, and to win around 99% of games against a low tier professional player.

I don’t remember anymore, but were those 7000 matches against high level players? Even the default bots on DotA2 can beat some median and below parties. And they’re very very bad and outdated.

Yes, openai 5 played OG who won the international (biggest dota 2 tournament in the world) 2 years in a row. Openai 5 beat them 2 games in a row in a best of 3

As far as I remember it all openai 5 games were not following standard DotA 2 rules of the time. Things like one courier per player instead of players fighting over a single one and the same heroes on both teams. Even I can win a game if I get to set the rules beforehand.

Agreed. Often I find reading the accompanying source code is more useful to understand the model.

> 3. What is the overall size of the model. Size, not parameter count.

What's the difference?

Reply on news.ycombinator.com