Inference Latency Evaluation on Qwen2.5-VL on NVIDIA Jetson Orin NX 8GB inference 0.5B
42.5Prefill LatencyEdgeFM
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| EdgeFMShape (Prefill/Decode Tokens)=512/322026.04 | 42.5 | -14.14 | 381.1 | -34.35 | 423.6 | -32.76 | |
| TRT-Edge-LLMShape (Prefill/Decode Tokens)=512/322026.04 | 49.5 | — | 580.5 | — | 630 | — | |
| EdgeFMShape (Prefill/Decode Tokens)=1024/322026.04 | 86.5 | -4.63 | 401 | -31.28 | 487.5 | -27.69 | |
| TRT-Edge-LLMShape (Prefill/Decode Tokens)=1024/322026.04 | 90.7 | — | 583.5 | — | 674.2 | — |