Join the community of Machine Learners and AI enthusiasts.
Sign Up
So my understanding is that the data used to train this model from the beginning is not English corpus, nor is it text, so its tokenizer is also different from the traditional one. I'm curious about how this part is handled and how the model itself understands things. Is it the same as the traditional one, which is also a one-dimensional token sequence?
Correct! It's causal modeling (for now) with a char level tokenizer with only 8 tokens.
The model learns by looking for relationships of sequences for a single token, so the only way it learns is literally nudging weights towards a generalized solution using pure sequences.
In short, it learns to learn.
Yes I am just diving into building my own AI models, I actually going to start learning to pretrain and fine tune model this this week. It a tone of thing to learn about AI.
actually thats a good idea pretraining the AI on a specific task I am just learning how to use contexts and system prompts prompt to guide the AI right now.
