Hacker News Reader

Comment by opnac | Hacker News Reader

I wish we could stop with the “X is all you need” papers! The first one was unintuitive and so are the rest.

I agree. X is all you need considered harmful

X is all you need considered harmful is all you need!

I agree that the copycats are wearing thin, but the original paper's title seems fine to me. It's an accurate description of the breakthrough they made. The first few sentences of the Attention is All You Need abstract explain it pretty well:

> The dominant sequence transduction models are based on complex recurrent or convolutional neural networks that include an encoder and a decoder. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.

https://proceedings.neurips.cc/paper/2017/file/3f5ee243547de...

It took a lot of my attention to even begin to understand that, but in the end, I agree with you.

The first sentence seems to imply that the main thing they want to do away with is the encoder–decoder arch, which they actually kept. (But BERT and GPT later did manage to simplify it.)

"is all you need" is considered harmful just like "considered harmful" is considered harmful by HNers?

Given that redundancy is considered harmful, you probably want to create a ConsideredHarmful class and then a ConsideredHarmfulFactory to make for a more enterprise-ready structure.

Thats so 2015, we need to move it to ConsideredHarmful Microservice

But I'm so close to finishing "goto is all you need"!

Beat me to it, I was going to go for "X Is All You Need Papers Considered Harmful"

And the dual of it, "X is harmful papers are all you need".

Didn't somebody prove the mov instruction is Turing-complete all by itself? I believe some code obfuscators actually take advantage of this.

At least in this case it's self-referential (not sure if intended)

There's this guy in Twitter fighting against titles like that: https://twitter.com/SayWhatYouFound

Huh, did twitter stop letting unauthenticated users read tweets? I can't get past login form.

You're one of today's lucky 10k: https://news.ycombinator.com/item?id=36540957

I agree. The paper is really interesting, but the title not so much :)

A bit click baity at least. And without opening it you have no chance to understand what this is about. I know HN has a policy against editorializing but in this case, a brief summary would have been helpful.

The paper introduces a new method for text generation, named Copy-Over-Generate (COG), which differs from traditional approaches that generate words from a fixed vocabulary. Instead, COG progressively copies phrases from a massive text collection, aiming to generate coherent text continuations through multiple rounds of phrase retrieval.

COG stands on the line of retrieval-augmented text generation research but takes a radical step forward. Unlike previous work that combines retrieval and generation, in COG, retrieval is generation.

COG shares some ideas with previous work such as replacing the fixed vocabulary with a nonparametric phrase table.

The paper presents experimental results showing the advantages of COG over strong baselines in three experimental settings: standard language modeling (using the WikiText-103 dataset), domain adaptation (using the Law-MT dataset), and an enlarged phrase index (using the En-Wiki dataset).

Despite the promising results, the authors acknowledge that there are some flaws in the COG method. For example, COG may copy a phrase that is incoherent with the previously copied phrase, or it may only copy a part of a complete phrase, leading to inaccurate generation results.

Before long, if these NN refinements continue at their current pace, it's going to become impossible to tell synthetic HN posts from organic ones. Going to get weird.

This is already the case with LLaMA.

Reply on news.ycombinator.com