jalammar

Hi HN,

This is the first in a series of articles I'm writing to introduce devs to practical applications of large NLP language models (for text generations like GPT and for language understanding like BERT).

I have been connecting the dots between the capabilities of these models and their business application. I still believe we're in the beginning of grasping the amount of potential value we can extract from these models. Happy to get to share these as I learn them from my exposure to the problem space.

Some of the key visual language I'm aiming to simplify is that of "prompts" and their use to shape model output (leading to practical applications). In this post, a key visual is [1] which shows an example of a summarization prompt and [2] showing a high-level process of "prompt engineering".

Would appreciate your feedback!

[1] https://docs.cohere.ai/img/intro-llms/language-model-prompt.... [2] https://docs.cohere.ai/img/intro-llms/prompt-engineering-and...

It's widely used in recommenders based on embeddings. See:

https://github.com/spotify/annoy

https://github.com/facebookresearch/faiss

Hi HN, I created this cheat sheet and video as high-level guidance to the major categories of ML explainability research and techniques. It's an area I've been working in the last year and this is how I synthesized the lay of the land in my head based on excellent references listed in the page. I hope you find it useful. It's non-exhaustive and will be updated, so please let me know if you have any feedback for future revisions. Thanks!

I wouldn't trust any model to generate text for customers yet. Not even the largest GPT3. There are no guarantees on what they will output and could be damaging to your business.

You're better off either: 1- Defining common "intents" that a lot of customer queries are categorized into, and having a model map the incoming message to the appropriate canned response. Look at Rasa, for an example of this.

2- if you insist on generating the text, have it be a recommendation to a human agent that either chooses to send it or writes their own response.

Hugging Face has that service

https://huggingface.co/pricing

I didn't come across one yet personally

Hello HN, author here. Language models are absolutely fascinating tools. I believe it would pay for software engineers to have a sense of their capabilities and how they function. The article showcases a few views to expose the inner workings of the model, but also simple UI for interacting with a language model to get a sense for how they work and generate words.

If you prefer video, I have also recently released a video [1] with PyData to provide an intro to language models and their applications and how we're trying to make Transformer-based ones more transparent with Ecco[2]. Contributors are welcome!

[1] https://www.youtube.com/watch?v=rHrItfNeuh0

[2] https://www.eccox.io/ and https://github.com/jalammar/ecco

Thanks mods for merging submissions. Happy to get feedback , thoughts, or questions.

Hey, I feel you! I'm an intuitive learner as well. I wouldn't have been able to learn much in ML if it weren't for people who write and visualize and make the methods accessible to non-experts. In my case, as with many others, it was the writing and videos of Andrew Ng, Karpathy, Chris Olah, Nando de Freitas, Sebastian Ruder, Andrew Trask, and Denny Britz amongst others. Accessible content like this goes a long way in building the confidence to further pursue the topic and not be intimidated by the steep learning curve. It fill me with joy that you've found some of my work helpful.

Thanks for digging up the screenshot. Exploring contextualize word embeddings is truly fascinating. And thanks for sharing your experience!

I actually started with PCA. But NMF proved more understandable since negative dimensions in PCA are hard to interpret. I didn't consider UMAP, but would be interested to see how it performs here.

It should be easy, yeah. for NMF, the activations vector is reshaped from (layers, neurons, token position) down into (layers/neurons, token position). And we present that to sklearn's NMF model. I would assume UMAP would operate on that same matrix. That matrix is called 'merged_act' and is located here: https://github.com/jalammar/ecco/blob/1e957a4c1c9bd49c203993...

Interesting. Thanks for sharing your notes on the higher layers. Allow me to repost that to the discussion board on github.

I do get your point on interpretation. This work is just a starting point. I'm curious to arrive at ways to automatically select the appropriate number of factors for a specific sequence. Kind of like the elbow method for K-means clustering.

These are AI explanation methods. They belong to the toolbox which would include LIME, Shapley values...etc. Input saliency is a gradient-based explanation method.

Wonderful! Thanks!

I am curious about those recent O(L) attention transformers (see slide 106 of http://gabrielilharco.com/publications/EMNLP_2020_Tutorial__...). If these methods are converging towards a new self-attention mechanism, I'd love to try illustrating that.

What other attention modes are you referring to? Did something in particular catch your attention?

I'd love to look at your group's visualizations! Is it a private repo? because the link doesn't open up. It never stops to blow my mind that we can represent words and concepts in vectors of numbers.

Thanks for your kind words! It's a labor of passion, honestly. And while in previous years it was a nights-and-weekends project, I have recently been giving it my entire time and focus -- which is why I'm able to dip my toes more heavily into R&D like Ecco and the "Explaining Transformers" article.

Thanks! Please let me know if you have any feedback!

What I found most fascinating is identifying neuron firing patterns corresponding to linguistic properties: e.g. groups of neurons that fire in response to verbs, or pronounds.

Scroll down to "Factorizing Activations of a Single Layer" in https://jalammar.github.io/explaining-transformers/ to see those.

The figure above it, titled 'Explorable: Ten Activation Factors of XML' shows how neuron firing patterns in response to XML -- opening tags, closing tags, and even indentation.

It's still fresh, but I'm keen to see what other people uncover in their examinations (or what shortfalls/areas of improvement there are for such a method).

Thank you!

Hi HN,

Author here. I had been fascinated with Andrej Karpathy's article (https://karpathy.github.io/2015/05/21/rnn-effectiveness/) -- especially where it shows neurons being activated in response to brackets and indentation.

I built Ecco to enable examining neurons inside Transformer-based language models.

You can use Ecco to simply interact with a language model and see its output token by token(as it's built on the awesome Hugging Face transformers package). But more interestingly you can use it to examine neuron activations. The article explains more: https://jalammar.github.io/explaining-transformers/

I have a couple more visualizations I'd like to add in the future. It's open source, so feel free to help me improve it.

Apple Keynote

Author here. Thank you. I feel an important element of this type of writing is what complexity to show and what to hide at different points. "Magic" is just to say "don't worry about the contents of this box yet, we'll get to it". It's what we discuss right after the visual. Sorry that came out as confusing. I'll add a note to the following figure saying that's the magic.

Hi HN,

This is the first video in a series where I gently explain AI/ML with an emphasis on applications rather than the math/stats/algorithms. It's meant for the general public, but I hope it helps the practitioners understand more about where the business applications are so they can optimize their careers or startups and specialize in domains that could benefit their careers.

It's also the first video on my channel. I hope to later create videos touching on other topics in my blog (https://jalammar.github.io/), like the Illustrated Transformers, BERT, Word2vec, and others.

I hope you enjoy it. All feedback is welcome!

Hi HN, author here. I'm hoping for this tutorial to be the gentlest way for you to get your first exposure to the BERT model. I hope you find it useful. All feedback/corrections are welcomed.

Thanks! Mostly Keynote, and lots of iteration.

I'll be honest, I personally found this figure puzzling. Still not 100% clear on it, but I don't believe it refers to the negative sampling approach. My best guess is that it's referring to earlier word2vec variants where the input in skipgram (or sum of inputs in CBOW) are multiplied by a weights matrix that projects the input to an output vector.

You're the first to point that one out! Nice catch!

Oh wow. Hi Radim! Huge fan of Gensim! Thanks for the links!

Very interesting. I'll read the paper to wrap my head around the concept. Thanks for the feedback!

Hello HN,

Author here. I wrote this blog post attempting to visually explain the mechanics of word2vec's skipgram with negative sampling algorithm (SGNS). It's motivated by:

1- The need to develop more visual language around embedding algorithms.

2- The need for a gentle on-ramp to SGNS for people who are using it for recommender systems. A use-case I find very interesting (there are links in the post to such applications)

I'm hoping it could also be useful if you wanted to explain to someone new to the field the value of vector representations of things. Hope you enjoy it. All feedback is appreciated!

Hello HN, author here. This post is my best attempt at visually explaining some of the leading NLP models that came up this year and some of the context surrounding them. Given BERT relies on the Transformer model, this post builds on my earlier post - The Illustrated Transformer https://jalammar.github.io/illustrated-transformer/

Hope you find it useful. Feedback is much appreciated!

Thanks for the heads up! Indeed it's df.loc[1:3] that would return three rows, not straight-up df[1:3] which indeed returns two rows.

Edit: Corrected in the post. Thanks again!

I haven't used tidyverse myself, but I know that pandas is heavily influenced and inspired by R. Most analysis tasks are doable in both platforms. If later stages of your pipeline involve deep learning (or machine learning, generally), then it could pay to be in the python universe given the wide adoption of python ML/DL tools. I generally wouldn't advise switching unless you have a certain pain point, though.

A reasonable next step would be pick up a dataset that interests you (in a domain you're comfortable with) and explore it with pandas. Kaggle has a bunch of data sets (https://www.kaggle.com/datasets) in various domains. You can look at the "Kernels" where other users often use pandas to uncover insights and show you their process.

Thanks for the kind words!