How much GPU memory does your model need?

Add up three things. Model weights: the number of parameters times bytes per parameter (2 at FP16, 1 at FP8 or INT8, 0.5 at INT4). KV cache: memory for each token in each active conversation, which grows with context length and users. Overhead: roughly 10% of the weights plus about 1 GB for the inference engine. The calculator above does this for you.

The weights alone take about 140 GB at FP16, 70 GB at FP8 and 35 GB at INT4. On top of that you need memory for the KV cache, which grows with context length and the number of users, plus some runtime overhead. A 70B model at FP8 serving 8 users with 8K context needs roughly 100 GB in total.

Kimi K3 has 2.8 trillion parameters (104B active per token) and is released only in 4-bit MXFP4, so the weights alone take roughly 1.5 TB. With overhead and a few users, plan for about 1.6 TB of GPU memory. That is more than one server of eight H200 GPUs holds, so it needs a server of eight 256 GB GPUs such as AMD Instinct MI325X, or two NVIDIA DGX B200 systems. Check that your inference engine supports MXFP4 on the GPUs you choose.

GLM-5.2 has 744B parameters with 40B active per token. In FP8 the weights take about 744 GB, so plan for roughly 790 GB with overhead and a few users: one 8-GPU server with NVIDIA H200 GPUs, or one NVIDIA DGX B200. The NVFP4 version is about 433 GB on disk and needs around 460 GB in total; NVFP4 runs natively on NVIDIA Blackwell GPUs such as B200. BF16 needs about 1.6 TB.

6 GB is enough only for a small 7B to 8B model at 4-bit with short prompts, for one person. 16 GB runs an 8B model at FP8 or a 14B model at 4-bit comfortably. 24 GB can fit a 32B model at 4-bit for a single user. Serving a team, longer documents or larger models needs data-center GPUs with 48 GB or more.

As a rule of thumb, at 4-bit you need about 0.6 GB of VRAM per billion parameters, at FP8 about 1.1 GB and at FP16 about 2.2 GB, plus extra for context and users. So a 48 GB GPU fits a 70B model at 4-bit for light use, and a 141 GB GPU fits it at FP8 for a small team. Choose your model size and precision above to check a specific setup.

Running a model on your own hardware keeps prompts, documents and outputs inside your building, which matters for confidential, regulated or client data. Costs are fixed rather than charged per token, the system can run offline or air-gapped, and you choose exactly which models are used and when they change.

Small models can run on a CPU, but responses are slow, so this suits testing rather than daily work. For a team using AI every day, a GPU with enough memory to hold the whole model is what makes responses fast enough to be useful.

System RAM should be at least as large as the model files so they can be loaded, and more if you run other software on the same server. For speed, though, what matters is GPU memory (VRAM): the model should fit entirely in VRAM.

Some tools can offload part of the model to system RAM and the CPU, but generation then becomes many times slower. Your options are a lower precision such as INT4, a smaller model, or splitting the model across several GPUs.

Start with memory: the GPU must hold the model and its KV cache. Then look at memory bandwidth, which largely decides how fast tokens are generated. For on-premise teams, data-center GPUs such as NVIDIA H200 or AMD Instinct MI300X hold large models on one card, while L40S or RTX PRO 6000 GPUs suit smaller models or can be combined in one server.

When a model generates text, it stores keys and values for every token in the conversation so it does not recompute them. That store is the KV cache. Every active user has their own, so memory grows with both context length and the number of people using the model at the same time.

FP8 and INT8 usually keep quality very close to FP16. INT4 cuts memory by about three-quarters and works well for many business tasks, but can lose some accuracy on complex reasoning. It is worth testing your own prompts before choosing.

Full fine-tuning stores the weights, their gradients and optimizer states, which adds up to about 16 bytes per parameter plus activations, so an 8B model needs over 130 GB. LoRA fine-tuning freezes the base model and trains small adapter layers, so it needs only a little more memory than running the model. Switch the workload above to compare.

It gives a planning estimate, not a guarantee. Real usage depends on the model architecture, the inference engine, batch settings and memory fragmentation. We assume a standard transformer with grouped-query attention and keep about 10% of each GPU free. Talk to us for a sizing check on your exact workload.