Decode Throughput on Llama 3B 3.2
117.3Throughput (tok/s)BaseRT
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| BaseRTQuantization=Q4, Hardware=Apple M4 Pro, Target Length=tg1282026.07 | 117.3 | 1.15 | 1.05 | |
| MLXQuantization=Q4, Hardware=Apple M4 Pro, Target Length=tg1282026.07 | 112.1 | — | — | |
| llama.cppQuantization=Q4, Hardware=Apple M4 Pro, Target Length=tg1282026.07 | 102.4 | — | — | |
| BaseRTQuantization=Q8, Hardware=Apple M4 Pro, Target Length=tg1282026.07 | 70.9 | 1.09 | 1.08 | |
| MLXQuantization=Q8, Hardware=Apple M4 Pro, Target Length=tg1282026.07 | 65.5 | — | — | |
| llama.cppQuantization=Q8, Hardware=Apple M4 Pro, Target Length=tg1282026.07 | 65.1 | — | — |