ResearchBenchmarksLarge Language Model Inference on Llama 3.1Large Language Model Inference on Llama 3.1Follow48.1Latency (ms)RoME47.99248.72149.4550.179Apr 10, 2026Evaluation resultsMethodMethodLinksLatency (ms)SpeedupRoMEModel=Llama3.1, Parame...Model=Llama3.1, Parameters=8B, QKShape[BNSD]=[1, 24, 8192, 128]2026.0448.14.7RoPEModel=Llama3.1, Parame...Model=Llama3.1, Parameters=8B, QKShape[BNSD]=[1, 24, 8192, 128]2026.0450.8—