LLM Inference Efficiency on Short sequence prompts
0.2TTFT (s)FedRAG
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| FedRAGModel=mistralai/Ministral-3-14B-Instruct-2512-BF16, Protected Layers=22026.05 | 0.2 | 33.77 | 173.14 | 75 | |
| FedRAGModel=openai/gpt-oss-20b, Protected Layers=22026.05 | 0.22 | 20.67 | 46 | 75 | |
| FedRAGModel=Qwen/Qwen3-4B-Instruct-2507, Protected Layers=22026.05 | 0.25 | 36.15 | 156.47 | 75 | |
| FedRAGModel=meta-llama/Llama-3.1-8B-Instruct, Protected Layers=22026.05 | 0.25 | 47.04 | 141.01 | 75 | |
| PermLLMModel=Qwen/Qwen3-4B-Instruct-2507, Protected Layers=22026.05 | 0.85 | 16.38 | 452.59 | 683 | |
| PermLLMModel=meta-llama/Llama-3.1-8B-Instruct, Protected Layers=22026.05 | 0.86 | 18.18 | 439.03 | 683 | |
| PermLLMModel=mistralai/Ministral-3-14B-Instruct-2512-BF16, Protected Layers=22026.05 | 0.93 | 19.68 | 466.64 | 558 | |
| PermLLMModel=openai/gpt-oss-20b, Protected Layers=22026.05 | 0.98 | 13.25 | 379.52 | 683 | |
| FedRAGModel=openai/gpt-oss-20b, Protected Layers=full2026.05 | 1.39 | 10.19 | 220.66 | 867 | |
| FedRAGModel=mistralai/Ministral-3-14B-Instruct-2512-BF16, Protected Layers=full2026.05 | 1.64 | 9.12 | 357.67 | 1,427 | |
| SCXModel=openai/gpt-oss-20b, Protected Layers=22026.05 | 3.13 | 17.79 | 941.4 | 484 | |
| FedRAGModel=meta-llama/Llama-3.1-8B-Instruct, Protected Layers=full2026.05 | 3.23 | 11.7 | 301.17 | 1,147 | |
| SCXModel=openai/gpt-oss-20b, Protected Layers=full2026.05 | 3.52 | 5.94 | 1,181.29 | 2,618 | |
| FedRAGModel=Qwen/Qwen3-4B-Instruct-2507, Protected Layers=full2026.05 | 3.65 | 9.91 | 328.01 | 1,287 | |
| SCXModel=meta-llama/Llama-3.1-8B-Instruct, Protected Layers=22026.05 | 4.86 | 30.97 | 1,526.88 | 580 | |
| SCXModel=Qwen/Qwen3-4B-Instruct-2507, Protected Layers=22026.05 | 5.2 | 25.66 | 1,418.02 | 628 | |
| SCXModel=Qwen/Qwen3-4B-Instruct-2507, Protected Layers=full2026.05 | 5.2 | 4.86 | 1,677.8 | 3,878 | |
| SCXModel=meta-llama/Llama-3.1-8B-Instruct, Protected Layers=full2026.05 | 6.22 | 5.13 | 1,791.25 | 3,458 | |
| SCXModel=mistralai/Ministral-3-14B-Instruct-2512-BF16, Protected Layers=full2026.05 | 9.06 | 4.07 | 2,406.17 | 4,298 | |
| SCXModel=mistralai/Ministral-3-14B-Instruct-2512-BF16, Protected Layers=22026.05 | 9.09 | 24.45 | 2,117.09 | 676 | |
| PermLLMModel=openai/gpt-oss-20b, Protected Layers=full2026.05 | 9.68 | 1.79 | 4,224.67 | 8,163 | |
| PermLLMModel=meta-llama/Llama-3.1-8B-Instruct, Protected Layers=full2026.05 | 11.93 | 1.62 | 4,721.73 | 10,851 | |
| PermLLMModel=Qwen/Qwen3-4B-Instruct-2507, Protected Layers=full2026.05 | 12.97 | 1.42 | 5,135.92 | 12,195 | |
| PermLLMModel=mistralai/Ministral-3-14B-Instruct-2512-BF16, Protected Layers=full2026.05 | 14.73 | 1.29 | 5,620.91 | 13,539 |