Text Generation Throughput on Synthetic Data
32.76Decode tok/sQwen2.5-0.5B, GGUF Q8_0
Evaluation results
| Method | Links | |
|---|---|---|
| Qwen2.5-0.5B, GGUF Q8_0Params=494M, Weight MB=506.5, RSS MB=565.0, Hardware=AMD Ryzen 7 5800X, CPU Threads=12026.06 | 32.76 | |
| SNN-0.8B, C++ INT8Params=874M, Weight MB=1087.4, RSS MB=1175.9, Hardware=AMD Ryzen 7 5800X, CPU Threads=12026.06 | 22.63 | |
| TinyLlama-1.1B, GGUF Q8_0Params=1.1B, Weight MB=1115.6, RSS MB=1071.7, Hardware=AMD Ryzen 7 5800X, CPU Threads=12026.06 | 16.31 | |
| Falcon3-1B, GGUF Q8_0Params=1.0B, Weight MB=1696.3, RSS MB=1487.1, Hardware=AMD Ryzen 7 5800X, CPU Threads=12026.06 | 11.26 | |
| Qwen2.5-1.5B, GGUF Q8_0Params=1.54B, Weight MB=1570.3, RSS MB=1634.4, Hardware=AMD Ryzen 7 5800X, CPU Threads=12026.06 | 9.7 |