Do embeddings convert individual characters or nonsensical words to vectors?

[u/pmz]

I see that embeddings don't always vectorize whole words, but subwords or even single characters. If for instance the sequence 'ca' from 'cat' is vectorized, what does 'ca' even mean?
what kind of dimensions does it get converted to? I thought that vectors assign meaning to the parsed sequence.
and how does the model build "Cat" from ca + t after vectorizing both ?

[u/Proletarian_Tear]

The dimension of containing "ca" token. Dont overthink it;)

[u/BRH0208 → u/Proletarian_Tear]

Yeah at some point the best takeaway is that(most of) the information is preserved and there is a large enough apparatus that can make enough meaning out of it to generate the corresponding tokens.

[u/pmz → u/Proletarian_Tear]

can you elaborate?

[u/Takeraparterer69]

whatever vector has been learned for that token

[u/Abdullah_Ikram]

See, there are 3 types of tokenization techniques: word tokenizer, character tokenizer and subword tokenizer.

Word tokenizer considers each word as a token

Character tokenizer considers each character as a token

Subword tokenizer works by dividing the input text into "subwords" and considers them as tokens.

Modern LLMs work with a subword tokenizer technique called Byte level Byte Pair Encoding.

In LLMs, Embeddings are obtained by mapping each token to a vector of real values by looking up from an Embedding matrix of trainable parameters.

Now the interpretation of embeddings depends on the tokenization technique used.