Hacker News Reader

Comment by amelius | Hacker News Reader

> where GPT is used to generate text, BERT is used to generate embeddings for input text that you can then use for predictive models (e.g. sentiment prediction)

Ok, but isn't text generation more general? E.g. you could ask it to predict the sentiment of a sentence and write the result as a sentence?

View on HN

Yeah my explanation was definitely a lossy summary. You can do similar things with GPT, but BERT is bidirectional, so for a given token it can take into account both tokens before and after it. GPT would only take into account tokens before it. Looking both ways can be helpful. Another comment in this thread explains the same (maybe clearer).

Yeah, he's glossing over some things, but with good reason. Might be more accurate to say BERT is a discriminant model, while GPT is a generative model. BERT was trained using Masked Language Model process, which is different from the decoder-only process used for the first GPT. Sentiment prediction seems like more of a particular thing BERT is capable of. There are many more capabilities, but GPT has sort of steered the industry towards generative models. https://huggingface.co/docs/transformers/main/tasks/masked_l...

GPT and BERT were actually the first models published after Attention was published by Google.

BERT is ~100m parameters. GPT3.5 is 1000x bigger. A Bert sized decoder model would likely generate nonsense

For most applications BERTs output is used to fine tune additional NN layers, eg for text classification

Reply on news.ycombinator.com