LLM Inference on MBPP sanitized
2.44Throughput (tokens/s)TIDE
Evaluation Results
| Method | Links | |
|---|---|---|
| TIDEModel=LLaDA2.0-mini, Gen Length=1024, GPU Expert Budget=128, GPU Memory Constraint=18 GB, Block Length=32, Confidence Threshold=0.952026.05 | 2.44 | |
| TIDEModel=LLaDA2.0-mini, Gen Length=256, GPU Expert Budget=128, GPU Memory Constraint=18 GB, Block Length=32, Confidence Threshold=0.952026.05 | 2.36 | |
| TIDEModel=LLaDA2.0-mini, Gen Length=256, GPU Expert Budget=64, GPU Memory Constraint=10 GB, Block Length=32, Confidence Threshold=0.952026.05 | 2.11 | |
| Mixtral-OffloadModel=LLaDA2.0-mini, Gen Length=1024, GPU Expert Budget=128, GPU Memory Constraint=18 GB, Block Length=32, Confidence Threshold=0.952026.05 | 1.91 | |
| TIDEModel=LLaDA2.0-mini, Gen Length=1024, GPU Expert Budget=64, GPU Memory Constraint=10 GB, Block Length=32, Confidence Threshold=0.952026.05 | 1.89 | |
| FiddlerModel=LLaDA2.0-mini, Gen Length=256, GPU Expert Budget=64, GPU Memory Constraint=10 GB, Block Length=32, Confidence Threshold=0.952026.05 | 1.81 | |
| FiddlerModel=LLaDA2.0-mini, Gen Length=1024, GPU Expert Budget=128, GPU Memory Constraint=18 GB, Block Length=32, Confidence Threshold=0.952026.05 | 1.8 | |
| FiddlerModel=LLaDA2.0-mini, Gen Length=1024, GPU Expert Budget=64, GPU Memory Constraint=10 GB, Block Length=32, Confidence Threshold=0.952026.05 | 1.79 | |
| Mixtral-OffloadModel=LLaDA2.0-mini, Gen Length=256, GPU Expert Budget=128, GPU Memory Constraint=18 GB, Block Length=32, Confidence Threshold=0.952026.05 | 1.76 | |
| FiddlerModel=LLaDA2.0-mini, Gen Length=256, GPU Expert Budget=128, GPU Memory Constraint=18 GB, Block Length=32, Confidence Threshold=0.952026.05 | 1.74 | |
| TIDEModel=LLaDA2.0-flash, Gen Length=256, GPU Expert Budget=64, GPU Memory Constraint=55 GB, Block Length=32, Confidence Threshold=0.952026.05 | 1.73 | |
| Mixtral-OffloadModel=LLaDA2.0-mini, Gen Length=256, GPU Expert Budget=64, GPU Memory Constraint=10 GB, Block Length=32, Confidence Threshold=0.952026.05 | 1.69 | |
| Mixtral-OffloadModel=LLaDA2.0-mini, Gen Length=1024, GPU Expert Budget=64, GPU Memory Constraint=10 GB, Block Length=32, Confidence Threshold=0.952026.05 | 1.45 | |
| TIDEModel=LLaDA2.0-flash, Gen Length=1024, GPU Expert Budget=64, GPU Memory Constraint=55 GB, Block Length=32, Confidence Threshold=0.952026.05 | 1.45 | |
| Mixtral-OffloadModel=LLaDA2.0-flash, Gen Length=256, GPU Expert Budget=64, GPU Memory Constraint=55 GB, Block Length=32, Confidence Threshold=0.952026.05 | 1.35 | |
| TIDEModel=LLaDA2.0-flash, Gen Length=256, GPU Expert Budget=32, GPU Memory Constraint=30 GB, Block Length=32, Confidence Threshold=0.952026.05 | 1.25 | |
| TIDEModel=LLaDA2.0-flash, Gen Length=1024, GPU Expert Budget=32, GPU Memory Constraint=30 GB, Block Length=32, Confidence Threshold=0.952026.05 | 1.24 | |
| Mixtral-OffloadModel=LLaDA2.0-flash, Gen Length=1024, GPU Expert Budget=64, GPU Memory Constraint=55 GB, Block Length=32, Confidence Threshold=0.952026.05 | 1.16 | |
| FiddlerModel=LLaDA2.0-flash, Gen Length=256, GPU Expert Budget=64, GPU Memory Constraint=55 GB, Block Length=32, Confidence Threshold=0.952026.05 | 1.14 | |
| FiddlerModel=LLaDA2.0-flash, Gen Length=1024, GPU Expert Budget=64, GPU Memory Constraint=55 GB, Block Length=32, Confidence Threshold=0.952026.05 | 1.05 | |
| Mixtral-OffloadModel=LLaDA2.0-flash, Gen Length=256, GPU Expert Budget=32, GPU Memory Constraint=30 GB, Block Length=32, Confidence Threshold=0.952026.05 | 1.01 | |
| Mixtral-OffloadModel=LLaDA2.0-flash, Gen Length=1024, GPU Expert Budget=32, GPU Memory Constraint=30 GB, Block Length=32, Confidence Threshold=0.952026.05 | 1.01 | |
| FiddlerModel=LLaDA2.0-flash, Gen Length=256, GPU Expert Budget=32, GPU Memory Constraint=30 GB, Block Length=32, Confidence Threshold=0.952026.05 | 0.95 | |
| FiddlerModel=LLaDA2.0-flash, Gen Length=1024, GPU Expert Budget=32, GPU Memory Constraint=30 GB, Block Length=32, Confidence Threshold=0.952026.05 | 0.89 |