EDGE OSDEFINITION
Tokenization is the process of splitting text into small pieces which are known as tokens. Tokens can be single words, sub-words, characters or punctuation marks, which depend on the model and tokenizer being used.
Within large language models, tokenization is the process of converting text, such as prompts from users, into numerical values representing tokens. This can affect the performance of your model, how long the context window is, how fast it processes and the cost of using your APIs.
Why is Tokenization important?
- Converts human languages to something AI models can understand.
- Helps models understand the structure of text.
- Affects how many tokens you use and the cost of processing with LLMs.
- Determines how much information can fit into the context window of your model.
Use cases
- Text processing within large language models.
- Chatbot and virtual assistant building.
- Text preprocessing for machine learning.
- Multilingual AI support and natural language processing.
See it running on your own hardware
Vanik Edge OS runs AI models, agents and apps inside your building, so your data stays there.