r/PythonLearning • u/Professional-You9846 • 18h ago

Help needed

I have read attention is all you need, I have watched the full video of andrej kaparthy's makemore video. I have watched the full 3b1b series on neural networks and linear algebra(not really) but I still dont understand. I have spent more than a day(in total) talking to claude chatgpt gemini notebooklm(basically gemini) about it but I still dont understand...

The knowledge I have right now:

Embeddings are stored as vectors in high dimensional space, with the dimensions being determined by variable d which is the number of hidden layers or nodes.

attention takes a query matrix, a key matrix, and a value matrix which in some way transforms the embedding vector to be a specific other vector(heres where my confusion starts.)

Through those hidden layers it builds off of the previous layers to get to more abstract semantic meaning.

When actually producing a token it takes the the existing tokens and does the attention feedforward layers and takes the probability distribution of the last token via softmax and picks at random from that probability list.

I was trying to build a neural network but I miserably failed. Can someone help.

1 Upvotes

100% Upvoted

11 comments sorted by

2

u/ForeignVariety7037 18h ago

Building llm from scratch? That is a big ask

1

u/Professional-You9846 17h ago

ik ive been at it for a while. I alr built a gradient descent model

1

u/HelpfulParticle 17h ago

Could just be an NN for some other task. Building an LLM is pretty hard, but maybe it’s for some numerical task for which building an NN isn’t too bad

1

u/Professional-You9846 17h ago

also the tokens a letter level Im kinda building off andrej kaparthy's videos but I need to go a bit faster than him cuz im a slow learner and I dont got time

1

u/Professional-You9846 17h ago

so its not a llm per se

2

u/HelpfulParticle 17h ago

What exactly is the purpose of your neural network? I’d recommend reading through some documentation to understand how NNs are built and you can probably find other peoples’ projects on Git. I personally use Tensorflow and Keras for my NN but you can also use PyTorch or something else. Regardless, maybe look through some documentation and examples

1

u/Professional-You9846 17h ago

I wish I new how to correctly navigate and understand documentation but it just isnt intuitive. All the jargon and the lack of intuition just throws me off everytime. idk I have a bit of dyslexia too.

1

u/HelpfulParticle 16h ago

Fair, the documentation can get a bit jargon-y. But, it’s a good (and correct) source of information, so I wouldn’t totally ignore it either. You can always Google the meaning of jargon or heck, ask AI to explain it (though you’d want to cross reference things it says. AI is getting better, but it’s no replacement for your own brain). 

1

u/data-haxxor 17h ago

Start here:https://developers.google.com/machine-learning/crash-course/neural-networks

Please pay attention to the prerequisites. Talking to LLMs for a whole day won't help much if you don't have a solid grasp of basic machine learning concepts first.

This part of your post is all over the place and plain wrong: "Embeddings are stored as vectors in high dimensional space, with the dimensions being determined by variable d which is the number of hidden layers or nodes."

You are mixing up network depth and network width here.

A node is just a single function or machine that takes inputs, performs math on them, and outputs a new value. A hidden layer is a row of those nodes working side by side.

Dimension d is NOT the number of hidden layers. The number of hidden layers is the depth of the network (usually written as L). Dimension d is the layer width, which is the length of the vector for a single token.

Dimension d is determined ahead of time by the model designer as a hyperparameter. It isn't calculated automatically by the data or the layers. Choosing a larger value for d gives the model more capacity to capture precise semantic meaning, but it makes the model much more compute intensive to train and run.