EDGE OSDEFINITION
Tokens per second is a metric used to measure how many tokens are being generated/processed by an AI model every second. It is often used to measure the performance of large language models.
A higher token rate typically means that an AI application will have a faster response rate. However, this can depend on many variables including model size, hardware, prompt length, batch size and serving configuration so should be considered along with TTFT and latency.
Importance of Tokens Per Second
- Measures how fast AI models can generate tokens.
- Helps measure how efficient LLMs are during inference.
- Can be used to compare different hardware and model configurations.
- Helps organizations to plan and provision their infrastructure.
Use Cases of Tokens Per Second
- Used to benchmark LLM performance.
- Used to measure how efficient your GPU and AI inference infrastructure is.
- Monitor how fast chatbots can generate responses.
- Optimize enterprise AI applications to increase throughput.
See it running on your own hardware
Vanik Edge OS runs AI models, agents and apps inside your building, so your data stays there.