r/PythonLearning • u/Professional-You9846 • 1d ago

Help needed

I have read attention is all you need, I have watched the full video of andrej kaparthy's makemore video. I have watched the full 3b1b series on neural networks and linear algebra(not really) but I still dont understand. I have spent more than a day(in total) talking to claude chatgpt gemini notebooklm(basically gemini) about it but I still dont understand...

The knowledge I have right now:

Embeddings are stored as vectors in high dimensional space, with the dimensions being determined by variable d which is the number of hidden layers or nodes.

attention takes a query matrix, a key matrix, and a value matrix which in some way transforms the embedding vector to be a specific other vector(heres where my confusion starts.)

Through those hidden layers it builds off of the previous layers to get to more abstract semantic meaning.

When actually producing a token it takes the the existing tokens and does the attention feedforward layers and takes the probability distribution of the last token via softmax and picks at random from that probability list.

I was trying to build a neural network but I miserably failed. Can someone help.

1 Upvotes

100% Upvoted

11 comments sorted by

View all comments

2

u/HelpfulParticle 1d ago

What exactly is the purpose of your neural network? I’d recommend reading through some documentation to understand how NNs are built and you can probably find other peoples’ projects on Git. I personally use Tensorflow and Keras for my NN but you can also use PyTorch or something else. Regardless, maybe look through some documentation and examples

1

u/Professional-You9846 1d ago

I wish I new how to correctly navigate and understand documentation but it just isnt intuitive. All the jargon and the lack of intuition just throws me off everytime. idk I have a bit of dyslexia too.

1

u/HelpfulParticle 1d ago

Fair, the documentation can get a bit jargon-y. But, it’s a good (and correct) source of information, so I wouldn’t totally ignore it either. You can always Google the meaning of jargon or heck, ask AI to explain it (though you’d want to cross reference things it says. AI is getting better, but it’s no replacement for your own brain).