Encrypted LLM Inference Execution Time on 128 token prompt
2.7Latency (s)PowerSoftmax
Evaluation results
| Method | Links | |
|---|---|---|
| PowerSoftmaxModel=Pythia 70M, #Heads=8, #Layers=62024.10 | 2.7 | |
| Taylor and GoldschmidtModel=Pythia 70M, #Heads=8, #Layers=62024.10 | 3.6 | |
| Square strategyModel=Pythia 70M, #Heads=8, #Layers=62024.10 | 7.42 | |
| PowerSoftmaxModel=Pythia 1.4B, #Heads=16, #Layers=242024.10 | 11.1 | |
| PowerSoftmaxModel=Llama 7B, #Heads=32, #Layers=322024.10 | 16.6 | |
| Taylor and GoldschmidtModel=Pythia 1.4B, #Heads=16, #Layers=242024.10 | 19.4 | |
| Square strategyModel=Pythia 1.4B, #Heads=16, #Layers=242024.10 | 42.1 | |
| Taylor and GoldschmidtModel=Llama 7B, #Heads=32, #Layers=322024.10 | 46.1 | |
| Square strategyModel=Llama 7B, #Heads=32, #Layers=322024.10 | 84.05 | |
| CeriumGPU=B200, Parallelization=8x2025.12 | 134 | |
| CeriumGPU=B200, Parallelization=4x2025.12 | 152 | |
| CeriumGPU=B200, Parallelization=2x2025.12 | 215 | |
| CeriumGPU=B200, Parallelization=1x2025.12 | 253 | |
| CeriumGPU=H100, Parallelization=8x2025.12 | 295 | |
| CeriumGPU=A100, Parallelization=8x2025.12 | 341 | |
| CeriumGPU=H100, Parallelization=4x2025.12 | 346 | |
| CeriumGPU=A100, Parallelization=4x2025.12 | 385 | |
| CeriumGPU=A100, Parallelization=2x2025.12 | 575 | |
| CeriumGPU=H100, Parallelization=2x2025.12 | 580 | |
| CeriumGPU=A100, Parallelization=1x2025.12 | 675 | |
| CeriumGPU=H100, Parallelization=1x2025.12 | 698 | |
| NexusGPU=A100, Parallelization=4x2025.12 | 13,271 |