EDGE OSDEFINITION
Caching is the ability to store frequently used data or results in a temporary storage/cache. This is useful as you can use the cached data rather than having to query it from somewhere such as a database, 3rd party service or resource intensive operation. Different types of data can be stored when using caching depending on your system, this could include model outputs, prompt prefixes, embeddings or frequently requested items.
Importance of Caching
- Latency: Improve response times
- Cost: Reduce queries to databases, API requests and AI inference costs.
- Scalability: Decrease strain on services.
- User Experience: Help improve consistent responses.
Use Cases
- Commonly requested product information in e-commerce sites.
- Common responses in customer support using AI.
- Embeddings for semantic search and retrieval augmented generation.
- LLM prompt caching to reduce common processing of supported prompt prefixes.
See it running on your own hardware
Vanik Edge OS runs AI models, agents and apps inside your building, so your data stays there.