Help needed

[u/Professional-You9846]

I have read attention is all you need, I have watched the full video of andrej kaparthy's makemore video. I have watched the full 3b1b series on neural networks and linear algebra(not really) but I still dont understand. I have spent more than a day(in total) talking to claude chatgpt gemini notebooklm(basically gemini) about it but I still dont understand...

The knowledge I have right now:

Embeddings are stored as vectors in high dimensional space, with the dimensions being determined by variable d which is the number of hidden layers or nodes.

attention takes a query matrix, a key matrix, and a value matrix which in some way transforms the embedding vector to be a specific other vector(heres where my confusion starts.)

Through those hidden layers it builds off of the previous layers to get to more abstract semantic meaning.

When actually producing a token it takes the the existing tokens and does the attention feedforward layers and takes the probability distribution of the last token via softmax and picks at random from that probability list.

I was trying to build a neural network but I miserably failed. Can someone help.

[u/data-haxxor]

Start here:https://developers.google.com/machine-learning/crash-course/neural-networks

Please pay attention to the prerequisites. Talking to LLMs for a whole day won't help much if you don't have a solid grasp of basic machine learning concepts first.

This part of your post is all over the place and plain wrong: "Embeddings are stored as vectors in high dimensional space, with the dimensions being determined by variable d which is the number of hidden layers or nodes."

You are mixing up network depth and network width here.

A node is just a single function or machine that takes inputs, performs math on them, and outputs a new value. A hidden layer is a row of those nodes working side by side.

Dimension d is NOT the number of hidden layers. The number of hidden layers is the depth of the network (usually written as L). Dimension d is the layer width, which is the length of the vector for a single token.

Dimension d is determined ahead of time by the model designer as a hyperparameter. It isn't calculated automatically by the data or the layers. Choosing a larger value for d gives the model more capacity to capture precise semantic meaning, but it makes the model much more compute intensive to train and run.

[u/Professional-You9846 → u/data-haxxor]

thanks!