LLM Inference Efficiency on Long sequence prompts
1.05TTFT (s)FedRAG
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| FedRAGModel=Qwen/Qwen3-4B-Instruct-2507, Protected Layers=22026.05 | 1.05 | 36.79 | 2,462.82 | 267 | |
| FedRAGModel=meta-llama/Llama-3.1-8B-Instruct, Protected Layers=22026.05 | 1.15 | 44.49 | 2,207.37 | 267 | |
| FedRAGModel=mistralai/Ministral-3-14B-Instruct-2512-BF16, Protected Layers=22026.05 | 1.29 | 31.59 | 2,719.49 | 267 | |
| FedRAGModel=openai/gpt-oss-20b, Protected Layers=22026.05 | 1.35 | 21.88 | 611.33 | 140 | |
| FedRAGModel=openai/gpt-oss-20b, Protected Layers=full2026.05 | 7.72 | 13.54 | 3,037.73 | 1,647 | |
| FedRAGModel=meta-llama/Llama-3.1-8B-Instruct, Protected Layers=full2026.05 | 12 | 11.85 | 4,238.54 | 4,219 | |
| FedRAGModel=mistralai/Ministral-3-14B-Instruct-2512-BF16, Protected Layers=full2026.05 | 12.51 | 9.17 | 5,042.98 | 5,267 | |
| FedRAGModel=Qwen/Qwen3-4B-Instruct-2507, Protected Layers=full2026.05 | 13.15 | 10.02 | 4,639.35 | 4,743 | |
| SCXModel=openai/gpt-oss-20b, Protected Layers=22026.05 | 36.42 | 18.06 | 14,331.9 | 1,060 | |
| SCXModel=openai/gpt-oss-20b, Protected Layers=full2026.05 | 39.13 | 6.01 | 17,274.69 | 9,530 | |
| SCXModel=Qwen/Qwen3-4B-Instruct-2507, Protected Layers=22026.05 | 52.82 | 26.22 | 21,620.4 | 1,204 | |
| SCXModel=Qwen/Qwen3-4B-Instruct-2507, Protected Layers=full2026.05 | 55.41 | 4.83 | 24,644.67 | 14,246 | |
| SCXModel=meta-llama/Llama-3.1-8B-Instruct, Protected Layers=22026.05 | 58.06 | 31.44 | 23,159.13 | 1,156 | |
| SCXModel=meta-llama/Llama-3.1-8B-Instruct, Protected Layers=full2026.05 | 60.15 | 5.5 | 26,202.25 | 12,674 | |
| PermLLMModel=mistralai/Ministral-3-14B-Instruct-2512-BF16, Protected Layers=22026.05 | 76.65 | 11.23 | 42,475.8 | 2,603 | |
| PermLLMModel=meta-llama/Llama-3.1-8B-Instruct, Protected Layers=22026.05 | 77.51 | 12.74 | 41,963.67 | 2,603 | |
| PermLLMModel=Qwen/Qwen3-4B-Instruct-2507, Protected Layers=22026.05 | 77.53 | 11.75 | 42,196.11 | 2,603 | |
| PermLLMModel=openai/gpt-oss-20b, Protected Layers=22026.05 | 77.91 | 10.24 | 40,677.89 | 2,603 | |
| SCXModel=mistralai/Ministral-3-14B-Instruct-2512-BF16, Protected Layers=22026.05 | 79.65 | 24.75 | 32,134.34 | 1,252 | |
| SCXModel=mistralai/Ministral-3-14B-Instruct-2512-BF16, Protected Layers=full2026.05 | 80.99 | 4.25 | 35,278.67 | 15,818 | |
| PermLLMModel=openai/gpt-oss-20b, Protected Layers=full2026.05 | 966.51 | 1.16 | 483,843.42 | 31,203 | |
| PermLLMModel=meta-llama/Llama-3.1-8B-Instruct, Protected Layers=full2026.05 | 1,243.09 | 0.96 | 587,393.74 | 41,571 | |
| PermLLMModel=Qwen/Qwen3-4B-Instruct-2507, Protected Layers=full2026.05 | 1,341.59 | 0.86 | 640,416.59 | 46,755 | |
| PermLLMModel=mistralai/Ministral-3-14B-Instruct-2512-BF16, Protected Layers=full2026.05 | 1,464.34 | 3.39 | 673,616.17 | 12,839 |