Language Modeling on WikiText-2 (test)
1.2PPLAWQ
Evaluation results
| Method | Links | ||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| AWQBackbone=Llama 2 13B, Context Length=2048, Quantization=2-bit weight-only PTQ2026.05 | 1.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 2.14 | — | — | |
| AWQBackbone=Llama 2 7B, Context Length=2048, Quantization=2-bit weight-only PTQ2026.05 | 2.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 2.14 | — | — | |
| FP16Type=None, Backbone=LLaMA-3-70B, Speedup=1.0x, Context Length=8192, Precision=16-bit2025.11 | 2.56 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BF16Model=Llama 3 70B, Bits=16.0, Context Length=81922024.06 | 2.59 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| baselineBackbone=Llama-3-70B, Avg. Num. Bits=162024.01 | 2.59 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| KVQuant-4bitBackbone=Llama-3-70B, Avg. Num. Bits=4.32-4.35, Outliers=1.0%, Quantization=NUQ2024.01 | 2.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QTIPModel=Llama 3 70B, Bits=4.00, Context Length=81922024.06 | 2.75 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QTIPType=vector, Backbone=LLaMA-3-70B, Speedup=1.7x, Context Length=8192, Precision=4-bit2025.11 | 2.75 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PAROQType=linear, Backbone=LLaMA-3-70B, Speedup=2.2x, Context Length=8192, Precision=4-bit2025.11 | 2.82 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FP16Precision=FP16, Model Architecture=L3-70B2026.04 | 2.85 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FP16Bits=16, Model=LLaMA-3, Size=70B, Group size=1282024.06 | 2.86 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FP16 BaselineMX Format=FP16, Model=LLaMA3-70B2024.11 | 2.86 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AWQType=linear, Backbone=LLaMA-3-70B, Speedup=2.4x, Context Length=8192, Precision=4-bit2025.11 | 2.96 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QuIP#Model=Llama 3 70B, Bits=4.00, Context Length=81922024.06 | 2.99 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QUIP#Type=vector, Backbone=LLaMA-3-70B, Speedup=1.9x, Context Length=8192, Precision=4-bit2025.11 | 2.99 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FP16Bits=16.0, Model=Llama-2 70B, Context Length=40962024.06 | 3.12 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| baselineBackbone=Llama-2-70B, Avg. Num. Bits=162024.01 | 3.12 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| KVQuant-4bitBackbone=Llama-2-70B, Avg. Num. Bits=4.32-4.35, Outliers=1.0%, Quantization=NUQ2024.01 | 3.13 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QTIPBits=4.0, Model=Llama-2 70B, Context Length=40962024.06 | 3.16 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QuIP#Bits=4.0, Model=Llama-2 70B, Context Length=40962024.06 | 3.18 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QTIPModel=Llama 3 70B, Bits=3.00, Context Length=81922024.06 | 3.18 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AQLMBits=4.0, Model=Llama-2 70B, Context Length=40962024.06 | 3.19 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QTIPBits=3.0, Model=Llama-2 70B, Context Length=40962024.06 | 3.26 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| W16A16Quantization=W16A16, Model Scale=LLaMA-2 70B2024.02 | 3.31 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Llama-2 (FP16)#Bit=16.00, Model Variant=Llama-2-70B2025.03 | 3.31 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FP16Model=LLaMA2-70B, Bits=16, Group Size=N/A2023.11 | 3.32 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FP16Model=LLaMA2-70B, Bit-width=16-bit, Quantization Method=None, Group Size=None2023.11 | 3.32 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BaselineModel=LLAMA-2 70B, Weight Quantization=None, Outlier Features=None, Sequence Length=20482024.03 | 3.32 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FP16Bits=16, Model=LLaMA-2, Size=70B, Group size=1282024.06 | 3.32 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DenseBackbone=Llama-2 70B, Sparsity=0%, Context Length=20482025.10 | 3.32 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| E-QATType=linear, Backbone=LLaMA-3-70B, Speedup=2.4x, Context Length=8192, Precision=4-bit2025.11 | 3.33 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QuIP#Bits=3.0, Model=Llama-2 70B, Context Length=40962024.06 | 3.35 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AQLMBits=3.0, Model=Llama-2 70B, Context Length=40962024.06 | 3.36 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| NF4-asymModel=LLaMA2-70B, Bits=4, Group Size=642023.11 | 3.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| NF4-symModel=LLaMA2-70B, Bits=4, Group Size=642023.11 | 3.42 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| NF4-asymModel=LLaMA2-70B, Bits=4, Group Size=1282023.11 | 3.42 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| INT4Model=LLaMA2-70B, Bits=4, Group Size=642023.11 | 3.44 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| NF4-symModel=LLaMA2-70B, Bits=4, Group Size=2562023.11 | 3.44 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| NF4-symModel=LLaMA2-70B, Bits=4, Group Size=1282023.11 | 3.44 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| NF4-asymModel=LLaMA2-70B, Bits=4, Group Size=2562023.11 | 3.44 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| INT4Model=LLaMA2-70B, Bits=4, Group Size=1282023.11 | 3.46 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| INT4Model=LLaMA2-70B, Bits=4, Group Size=2562023.11 | 3.49 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| NF4-asymModel=LLaMA2-70B, Bits=4, Group Size=12023.11 | 3.51 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| NF4-symModel=LLaMA2-70B, Bits=4, Group Size=12023.11 | 3.52 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FP16Bits=16.0, Model=Llama-1 65B, Context Length=20482024.06 | 3.53 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| baselineModel=LLaMA-65B, Precision=fp16, KV Cache Size (GB)=3202024.01 | 3.53 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FP16W-Bits=16, Model Family=LLaMA-1, Model Size=65B2025.05 | 3.53 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| KVQuant-4bit-1%Model=LLaMA-65B, Bit-width=4-bit, Outlier Ratio=1%, KV Cache Size (GB)=86.52024.01 | 3.54 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| KVQuant-4bitModel=LLaMA-65B, Bit-width=4-bit, KV Cache Size (GB)=802024.01 | 3.55 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FlexGen-4bitModel=LLaMA-65B, Bit-width=4-bit, KV Cache Size (GB)=86.32024.01 | 3.56 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ATOM-4bitModel=LLaMA-65B, Bit-width=4-bit, KV Cache Size (GB)=83.12024.01 | 3.57 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| KVQuant-3bit-1%Model=LLaMA-65B, Bit-width=3-bit, Outlier Ratio=1%, KV Cache Size (GB)=66.52024.01 | 3.57 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QTIPBits=4.0, Model=Llama-1 65B, Context Length=20482024.06 | 3.58 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QuIP#Model=Llama 3 70B, Bits=3.00, Context Length=81922024.06 | 3.59 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QuIP#Bits=4.0, Model=Llama-1 65B, Context Length=20482024.06 | 3.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QuaRot-128GModel=LLAMA-2 70B, Weight Quantization=GPTQ-128G, Outlier Features=0, Group Size=128, Sequence Length=20482024.03 | 3.61 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| nf4Model=LLaMA-65B, Bit-width=4-bit, KV Cache Size (GB)=80.12024.01 | 3.63 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| KVQuant-3bitModel=LLaMA-65B, Bit-width=3-bit, KV Cache Size (GB)=602024.01 | 3.63 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPTVQBits=4.13, Model=Llama-1 65B, Context Length=20482024.06 | 3.64 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ShiftAddLLMBits=3, Model=LLaMA-2 70B2024.06 | 3.64 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ShiftAddLLM (Acc.)Bits=3, Model=LLaMA-2, Size=70B, Group size=1282024.06 | 3.64 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FlexGen-3bitModel=LLaMA-65B, Bit-width=3-bit, KV Cache Size (GB)=65.92024.01 | 3.66 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| INT4Model=LLaMA2-70B, Bits=4, Group Size=12023.11 | 3.67 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QTIPBits=3.0, Model=Llama-1 65B, Context Length=20482024.06 | 3.68 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QuIP#Bits=3.0, Model=Llama-1 65B, Context Length=20482024.06 | 3.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QTIPBits=2.0, Model=Llama-2 70B, Context Length=40962024.06 | 3.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| KVQuant-2bit-1%Model=LLaMA-65B, Bit-width=2-bit, Outlier Ratio=1%, KV Cache Size (GB)=46.52024.01 | 3.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ClusComp#Bit=2.72, Model Variant=Llama-2-70B2025.03 | 3.72 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| int4Model=LLaMA-65B, Bit-width=4-bit, KV Cache Size (GB)=80.12024.01 | 3.73 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AFPQ (NF3-asym)Model=LLaMA2-70B, Bit-width=3-bit, Quantization Method=RTN, Group Size=g642023.11 | 3.74 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AWQBits=3, Model=LLaMA-2, Size=70B, Group size=1282024.06 | 3.74 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SqueezeLLM#Bit=3.02, Model Variant=Llama-2-70B2025.03 | 3.77 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ATOM-3bitModel=LLaMA-65B, Bit-width=3-bit, KV Cache Size (GB)=632024.01 | 3.78 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPTVQBits=3.13, Model=Llama-1 65B, Context Length=20482024.06 | 3.79 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QuaRotModel=LLAMA-2 70B, Weight Quantization=GPTQ, Outlier Features=0, Sequence Length=20482024.03 | 3.79 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AFPQ (NF3-asym)Model=LLaMA2-70B, Bit-width=3-bit, Quantization Method=RTN, Group Size=g1282023.11 | 3.82 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AQLMBits=2.0, Model=Llama-2 70B, Context Length=40962024.06 | 3.83 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FPModel=Mixtral-8×7B (46.7B - A12.9B), #Bits=16.02026.05 | 3.84 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FP#Bits=16.0, Model=Mixtral-8×7B2026.05 | 3.84 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| INT3Model=LLaMA2-70B, Bit-width=3-bit, Quantization Method=RTN, Group Size=g642023.11 | 3.85 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| NF3-symModel=LLaMA2-70B, Bit-width=3-bit, Quantization Method=RTN, Group Size=g642023.11 | 3.85 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FP16Model=MIXTRAL-8X7B, Quantization=2-bit2026.05 | 3.87 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FP16Model=MIXTRAL-8X7B, Quantization=3-bit2026.05 | 3.87 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OPTQBits=3, Model=LLaMA-2, Size=70B, Group size=1282024.06 | 3.88 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AFPQ (NF3-asym)Model=LLaMA2-70B, Bit-width=3-bit, Quantization Method=RTN, Group Size=g2562023.11 | 3.89 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QuIP#Bits=2.0, Model=Llama-2 70B, Context Length=40962024.06 | 3.91 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| NF3-symModel=LLaMA2-70B, Bit-width=3-bit, Quantization Method=RTN, Group Size=g1282023.11 | 3.92 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FlexRoundBits=3, Model=LLaMA-2 70B2024.06 | 3.92 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BTC-LLMW-Bits=1.11, Model Family=LLaMA-1, Model Size=65B2025.05 | 3.94 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| INT3Model=LLaMA2-70B, Bit-width=3-bit, Quantization Method=RTN, Group Size=g1282023.11 | 3.98 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| NF3-symModel=LLaMA2-70B, Bit-width=3-bit, Quantization Method=RTN, Group Size=g2562023.11 | 3.98 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LUT-GEMMBits=3, Model=LLaMA-2, Size=70B, Group size=1282024.06 | 4.01 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VPTQW-Bits=2, Model Family=LLaMA-1, Model Size=65B2025.05 | 4.01 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| KVQuant-2bitModel=LLaMA-65B, Bit-width=2-bit, KV Cache Size (GB)=402024.01 | 4.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BTC-LLMW-Bits=0.90, Model Family=LLaMA-1, Model Size=65B2025.05 | 4.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FP16Bits=16.0, Model=Llama-1 30B, Context Length=20482024.06 | 4.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| baselineModel=LLaMA-30B, Precision=fp16, KV Cache Size (GB)=1952024.01 | 4.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OriginalModel=Llama-30B2026.02 | 4.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FP16W-Bits=16, Model Family=LLaMA-1, Model Size=30B2025.05 | 4.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TILEQsModel=MIXTRAL-8X7B, Quantization=3-bit, Extra Bits (Scales/Low-rank)=0.16/0.032026.05 | 4.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |