On a 24 GB VRAM GPU with a 7B FP16 model (14 GB weights) and each user requiring 800 MB KV cache, how many users can be served before OOM crash?

Question

Accepted Answer

The scaling table shows: 8 users → 14 + 6.4 = 20.4 GB, leaving 3.6 GB free (tight but OK). 16 users → 14 + 12.8 = 26.8 GB > 24 GB → OOM crash.