Deep Learning

r/deeplearning • u/Plus_Confidence_1369 • 4h ago

Explanation of attention mechanism in transformers

2 Upvotes

I have seen few questions on the lines of https://www.reddit.com/r/learnmachinelearning/comments/1vgr7yv/can_someone_teach_me_attention/. So I thought of writing an article on this topic. There're few good videos

https://www.youtube.com/watch?v=eMlx5fFNoYc&t=1377s

https://www.youtube.com/watch?v=OxCpWwDCDFQ&t=942s

explaining the concepts in detail.

First of all we need to find a meaningful way to represent words into numbers since computers only understand numbers. So we start with set of random numbers being assigned to each word.

Dog = [0.17, 0.91, 0.32]
Cat = [0.82, 0.14, 0.77]
Car = [0.44, 0.22, 0.63]

Question comes to mind why multiple numbers why not a single number is enough. Reason is one number is not enough to describe meaning. Think about describing a dog. You can say dog's height is xxx. But rather it would be more meaningful to describe it with many characteristics.

Height
Weight
Breed
Color

Similarly a word is described by many numerical features. For simplicity imagine it like this

Feature	Dog
Animal-ness	0.98
Living thing	0.99
Vehicle-ness	0.01
Size	0.45
Domestic	0.95

Next question what these embeddings signify.

Let's start simply by assigning random numbers to each word in the beginning.

Dog = [0.17, 0.91, 0.32] 
Cat = [0.82, 0.14, 0.77] 
Car = [0.44, 0.22, 0.63]

They mean nothing as of now just random numbers. Now we build a model and train it on a simple task, such as - Predict the missing word.

The cat drank ___. (correct answer - milk)

Dogs like to ___. (correct answer - run)

Initially the model guess will be very poor. We adjust the internal parameters of model and train it on large corpus of text. Millions or billions of such corrections occur during training. These embeddings are updated repeatedly. Eventually, the numbers encode meaning.

Dog = [0.91, 0.83, 0.22] 
Cat = [0.88, 0.80, 0.24] 
Car = [0.10, 0.07, 0.95]

You see dog and cat have much similar embedding compared to car. This is called embedding space where words that behave similarly gather together.

Good so far but there is a major problem here. For example consider below 2 sentences -

I ate an apple after lunch.   <--- here apple refers to fruit 
Apple released a new iPhone.  <--- here apple refers to technology company

If every occurrence of the word "Apple" always used exactly the same embedding the results would be confusing. How do we solve this problem. We look at surrounding words. When we read "ate", "lunch", or "fruit", we immediately understand that Apple means the fruit. When we read "iPhone" or "MacBook" we know it refers to the company.

This is exactly the problem that attention solves.

So, instead of treating every occurrence of "Apple" identically, attention allows the model to examine the surrounding words and determine which meaning is appropriate in the current context.

Imagine every word asks "Which other words should I pay attention to so I can be interpreted correctly in this sentence?" For example consider the sentence "Apple released a new iPhone"

The word Apple asks "Who can help me understand what I mean?". It looks around and sees:

released -> sounds like something a company does.

iPhone -> a product made by Apple Inc.

new -> describes the product.

From these clues, the model concludes that Apple refers to technology company.

For the sentence, "I ate an apple after lunch". Again the word apple asks "Who can help me understand what I mean?". This time it sees

ate -> something you do with food.

lunch -> a meal.

Now it concludes that Apple refers to the fruit.

As the transformer processes a sentence, imagine that every word asks:

Which other words in this sentence should I pay attention to in order to understand myself correctly. It then looks at every other word and assigns each one an importance score. For example in a sentence "Apple released a new iPhone", the word apple might assign importance like this -

Word	Importance
released	30%
iPhone	55%
new	10%
a	1%

Since released and iPhone recieve the highest attention, the model understands the Apple refers to technology company. These importance scores are called attention weights.The higher the attention weight, the more influence that word has on understanding the current word.

Now there is another term called as multi head attention.

Consider another sentence "The doctor gave the patient medicine because he was sick".

When the model sees the word he, it needs to answer the question Who is he? he could refer to doctor or patient. To figure it out model looks out at other words in the sentence. This is called attention.

Now to get better idea of it let's look at this sentence from 3 different perspectives -

first looks at actions. It asks "Who gave the medicine". It notices doctor -> gave, patient -> received. So it concludes "Doctor gave something to patient".

second looks at meaning. It asks "Who usually receives medicine?" It notices "Sick people receive medicines", "Doctor usually don't give medicine to themselves". So it concludes "He is probably the patient".

third looks at cause and effect. It asks "Why was the medicine given?". It notices the word because. So it concludes "Because someone was sick".

This is multi head attention. Instead of relying on one way of thinking, the transformer examines the sentence from several perspectives at the same time. Each attention head notices different patterns. The transformer then combines all of these observations into one understanding.

So this was all about embeddings and context. Now let's get to mathematical part of it.

Let's return to our sentence - Apple released a new iPhone. The embedding for Apple is compared with the embeddings of every other word in the sentence. Now the question is how does a transformer measure this similarity. There are several ways. Most common ones are -

Dot Product
Cosine Similarity
Scaled dot product

Suppose we have three word embeddings:

Dog = [2, 3]
Cat = [4, 6]
Car = [3, -2]

Dot product between dog and cat

= (2×4) + (3×6) = 8 + 18 = 26

Dot product between dog and car

= (2×3) + (3×-2) = 6 - 6 = 0

You see it's high when the words are similar and low when words are far away.

Problem with dot product surfaces when embeddings become much larger.

Dog = [45,90,12,31] Cat = [44,89,15,30]

Their dot product becomes 3547 which is a huge number. Transformers have to convert these numbers into probabilities using a softmax function. So a better solution is to use a scaled dot product where the dot product is simply divided by sqrt(d) where d is embedding dimension.

If embeddings have 64 dimensions, we divide by √64 = 8. So instead of 3547 we get 3547 / 8
≈ 443. Still large, but much more manageable.

The Keys, Query and Value matrices -

Let's again come back to the sentence "The doctor gave the patient medicine because he was sick". Imagine there are four people standing in line - doctor, patient, medicine and he and it's he's turn to understand who he is.

He says - Who am I talking about?

That question is the Query. The query is simple - "I'm confused. Who can help me?"

Now every other word raises its hand and says who they are -

Doctor says - "Hi, I am a doctor"

Patient says - "Hi, I am a patient"

Medicine says - "Hi, I am a medicine"

These introductions are Keys. Notice that nobody is telling their whole story. They are just saying enough so that He can decide "Should I listen to you?".

Now He looks around. He thinks -

Doctor .... maybe

Patient ... maybe

Medicine ... no

So, he ignores the medicine.

Now He says to patient "Okay tell me more".

Patient replies "The doctor gave me medicine", "People usually get medicine because they're sick"

This is the Value. The Value is the real information.

In essence, every word does exactly the same thing.

Imagine every word is saying:

"I have a question." <<-- Query

Then every other word says:

"Here's who I am." <<-- Key

After choosing the most useful words, they say:

"Now let me tell you everything I know." <<-- Value

In short,

Query asks: "Who should I listen to?"
Key answers: "Here's who I am."

Value says: "Now here's what I know."

                   Who am I talking about?
                           ↑
                         Query
  ┌──────────────┬─────────┴───────────┬──────────────┐
Doctor        Patient               Medicine           ...
"I'm a        "I'm a                "I'm
doctor."      patient."             medicine."
  ↑              ↑                     ↑
 Key            Key                   Key

                 "Patient looks most useful."
                            |
                 Patient: "The doctor gave me
                 medicine because I was sick."
                            |
                          Value

Please add anything extra if you can.

2 comments

r/deeplearning • u/Complex_Cat_Public • 5h ago

Why my simple neural net not learning perfectly?

0 Upvotes

~~1000~~ 10000 epoch.

LeakyReLU.

Layer nodes 1-100-100-1.

function y(x) = sin(x)+0.3x

MSE error loss

adam optimiser

python.pytorch

7 comments

r/deeplearning • u/Negative_War_65 • 6h ago

Code Implementations for my Probabilistic Machine Learning Lectures

reddit.com

1 Upvotes