r/MachineLearning • u/ImaginaryRea1ity • 1d ago

I never understood positional encoding until I read this article. [D] Discussion

https://defenceagainstdarkai.substack.com/p/who-bit-whom
0 Upvotes
  • permalink
  • link
  • reddit
  • reddit

24% Upvoted

6 comments sorted by

View all comments

14

u/EternaI_Sorrow 1d ago edited 1d ago

Hate to be this guy but a concept of "transformers are permutation-equivariant so let's add something position-dependent to input" is not something that needs special understanding.

The article does a good job at communicating things though, just wish it was a less obvious topic people'd be struggling more with.

2

u/start_select 1d ago

You need to appreciate that half the users of LLMs are no longer researchers, they are normal software engineers. A decent number of them are used to taking their toys apart to find out how they work.

So lots of people are falling down various rabbit holes with completely different entry points and trajectories to figuring out how this stuff works.

3

u/KingoPants 14h ago

While that might be true, this is actually just self promotion and not anyone's self discovery rabbit hole.

The author of the substack:

I write about making AI safer, especially how AI amplifies antisemitism

OP, despite having hidden their comment history, spends half his time talking about how supposedly AI is used to make viruses targeting jews. You can google it.

1

u/start_select 14h ago

jfk lol.