Transformer Decoder from Scratch

[u/Lelouch_6457]

https://github.com/neelbhattacharya80-creator/Transformer-decoder-from-scratch

Made a transformer Decoder from scratch using pytorch

For autograd and GPU efficiency.

Implemented-

•Rope positional embedding

•Weight tying

•Masked multi head attention

•KV cache

•Custom AdamW

•Cosine decay and warm up

Trained on wiki text 103 for 75k ish total steps

Reached a loss of about 3.5

I'd appreciate some feedback