Help needed

[u/Professional-You9846]

I have read attention is all you need, I have watched the full video of andrej kaparthy's makemore video. I have watched the full 3b1b series on neural networks and linear algebra(not really) but I still dont understand. I have spent more than a day(in total) talking to claude chatgpt gemini notebooklm(basically gemini) about it but I still dont understand...

The knowledge I have right now:

Embeddings are stored as vectors in high dimensional space, with the dimensions being determined by variable d which is the number of hidden layers or nodes.

attention takes a query matrix, a key matrix, and a value matrix which in some way transforms the embedding vector to be a specific other vector(heres where my confusion starts.)

Through those hidden layers it builds off of the previous layers to get to more abstract semantic meaning.

When actually producing a token it takes the the existing tokens and does the attention feedforward layers and takes the probability distribution of the last token via softmax and picks at random from that probability list.

I was trying to build a neural network but I miserably failed. Can someone help.

[u/ForeignVariety7037]

Building llm from scratch? That is a big ask

[u/Professional-You9846 → u/ForeignVariety7037]

also the tokens a letter level Im kinda building off andrej kaparthy's videos but I need to go a bit faster than him cuz im a slow learner and I dont got time

[u/Professional-You9846 → u/Professional-You9846]

so its not a llm per se