ResearchTasksLLM Inference LatencyLLM Inference LatencyFollowBenchmarksDatasetSOTA methodSortMost resultsRecently updatedMost papersApplyDatasetSOTA methodMetricTrendResultsLast updated256 tokens generation (test)All (Predictor, Windowing, Bundling)26I/O Latency (ms)21Feb 26, 2026Multi-node InfiniBand cluster (2x8-GPU servers) inference performance (test)FoE0.939Time To First Token (TTFT) Mean (s)4May 8, 2026
Multi-node InfiniBand cluster (2x8-GPU servers) inference performance (test)FoE0.939Time To First Token (TTFT) Mean (s)4May 8, 2026