5950x is a CPU model. Integer-quantized models are generally run with CPU inference. For the larger models the problem then becomes generation time per token.

Quantized models are used aplenty with GPUs as well - 4-bit quantization is the only way you can squeeze llama-30b into 24Gb of VRAM (i.e. RTX 3090 or 4090).

In fact, I would say that, at this point, most people running LLaMA locally are likely using 4-bit quantization regardless of model size and hardware, just to get the most out of the latter.

Most people running llama locally are doing CPU inference, period.

If your desktop had 256gb of ram, could you train a far larger model? Some motherboards support that