EDGE OSDEFINITION
Quantization is a technique that stores an AI model’s parameters with fewer bits, for example 8-bit or 4-bit numbers instead of 16-bit ones. This shrinks the memory a model needs and usually speeds up inference, at the cost of a small loss in accuracy that grows as precision drops. Common formats include FP8, INT8, INT4 and 4-bit floating-point formats such as NVFP4 and MXFP4.
Planning hardware for AI on your own premises? Work out the GPU memory a model needs in our VRAM calculator, or let us size it with you.
Get a free sizing checkWhy Quantization Matters
- Memory: Cuts GPU memory needs in half at 8-bit and by about three-quarters at 4-bit.
- Cost: Lets larger models run on fewer or smaller GPUs.
- Speed: Moves less data for every token, which often increases tokens per second.
- Quality: 8-bit formats usually stay very close to full-precision results.
Use Cases
- Running large open-weight models on-premise within limited GPU memory.
- Serving more concurrent users from the same hardware.
- Deploying models on workstations and edge devices.
- QLoRA fine-tuning, which trains small adapters on top of a 4-bit base model.
See it running on your own hardware
Vanik Edge OS runs AI models, agents and apps inside your building, so your data stays there.