Decode Throughput on Llama 3.2 1B
295.4Throughput (TOK/s)BaseRT
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| BaseRTQuantization=Q4, Hardware=Apple M4 Pro, Target Length=tg1282026.07 | 295.4 | 1.28 | 1.15 | |
| MLXQuantization=Q4, Hardware=Apple M4 Pro, Target Length=tg1282026.07 | 257.8 | — | — | |
| llama.cppQuantization=Q4, Hardware=Apple M4 Pro, Target Length=tg1282026.07 | 230.4 | — | — | |
| BaseRTQuantization=Q8, Hardware=Apple M4 Pro, Target Length=tg1282026.07 | 183.8 | 1.14 | 1.15 | |
| llama.cppQuantization=Q8, Hardware=Apple M4 Pro, Target Length=tg1282026.07 | 160.7 | — | — | |
| MLXQuantization=Q8, Hardware=Apple M4 Pro, Target Length=tg1282026.07 | 159.2 | — | — |