Multi-agent trajectory simulation on Waymo Open Sim Agents Challenge (WOSAC) 2024 (test)
1.2947minADEUniMM
Evaluation results
| Method | Links | |||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| UniMM2024.12 | 1.2947 | — | — | — | — | — | — | 96.79 | 0.8347 | — | 95.06 | — | — | — | — | — | 0.7683 | 0.3836 | 0.4159 | 0.5168 | 0.6491 | 0.3911 | 0.6791 | |
| SMART-tiny RoaD# model params=7 M2025.12 | 1.3042 | — | — | — | — | — | — | — | — | — | — | 0.7847 | — | 0.4932 | 0.8106 | 0.9178 | — | — | — | — | — | — | — | |
| SMART-tiny CAT-K# model params=7 M2025.12 | 1.3065 | — | — | — | — | — | — | — | — | — | — | 0.7846 | — | 0.4931 | 0.8106 | 0.9177 | — | — | — | — | — | — | — | |
| SMART-tiny fine-tuned w. CAT-K# model params=7 M2024.12 | 1.3068 | — | — | — | — | — | — | — | — | — | — | 0.7702 | 0.0088 | 0.4931 | 0.8119 | 0.8749 | — | — | — | — | — | — | — | |
| SMART-tiny-CLSFTTraining=Closed-Loop Supervised Fine-Tuning2024.12 | 1.3068 | — | — | — | — | — | — | 97.02 | 0.8356 | — | 95.23 | — | — | — | — | — | 0.7702 | 0.3868 | 0.4066 | 0.5201 | 0.6589 | 0.3923 | 0.6814 | |
| SMART-tiny (reproduced)Status=reproduced, Training=Behavioral Cloning (BC)2024.12 | 1.3587 | — | — | — | — | — | — | 96.53 | 0.8346 | — | 95.07 | — | — | — | — | — | 0.7671 | 0.3781 | 0.4026 | 0.5183 | 0.6571 | 0.3899 | 0.6788 | |
| SMART-large# model params=102 M2024.12 | 1.3728 | — | — | — | — | — | — | — | — | — | — | 0.7614 | 0 | 0.4786 | 0.8066 | 0.8648 | — | — | — | — | — | — | — | |
| SMART-largeBackbone=Large2024.12 | 1.3728 | — | — | — | — | — | — | 96.32 | 0.8346 | — | 94.03 | — | — | — | — | — | 0.7614 | 0.3786 | 0.4134 | 0.4952 | 0.627 | 0.3872 | 0.6761 | |
| SMART-large# model params=102 M2025.12 | 1.3728 | — | — | — | — | — | — | — | — | — | — | 0.7614 | — | 0.4786 | 0.8066 | 0.8648 | — | — | — | — | — | — | — | |
| SMART-tiny# model params=7 M2024.12 | 1.4062 | — | — | — | — | — | — | — | — | — | — | 0.7591 | -0.0023 | 0.4759 | 0.8039 | 0.8632 | — | — | — | — | — | — | — | |
| SMART-tinyBackbone=Tiny2024.12 | 1.4062 | — | — | — | — | — | — | 96.01 | 0.8338 | — | 94.01 | — | — | — | — | — | 0.7591 | 0.3733 | 0.4082 | 0.4945 | 0.6277 | 0.3835 | 0.6709 | |
| SMART-tiny# model params=7 M2025.12 | 1.4062 | — | — | — | — | — | — | — | — | — | — | 0.7591 | — | 0.4759 | 0.8039 | 0.8632 | — | — | — | — | — | — | — | |
| BehaviorGPT#Param=3M2024.05 | 1.4147 | 0.7473 | 0.3615 | 0.3365 | 0.4806 | 0.5544 | 0.3834 | 95.37 | 0.8308 | 0.6702 | 93.49 | — | — | — | — | — | — | — | — | — | — | — | — | |
| BehaviorGPT# model params=3 M2024.12 | 1.4147 | — | — | — | — | — | — | — | — | — | — | 0.7473 | -0.0141 | 0.4333 | 0.7997 | 0.8593 | — | — | — | — | — | — | — | |
| BehaviorGPT2024.12 | 1.4147 | — | — | — | — | — | — | 95.37 | 0.8308 | — | 93.49 | — | — | — | — | — | 0.7473 | 0.3615 | 0.3365 | 0.4806 | 0.5544 | 0.3834 | 0.6702 | |
| KIGRAS2024.12 | 1.4383 | — | — | — | — | — | — | 96.19 | 0.8373 | — | 94.31 | — | — | — | — | — | 0.7597 | 0.3704 | 0.3784 | 0.4962 | 0.6314 | 0.3867 | 0.6723 | |
| KIGRAS# model params=0.7 M2024.12 | 1.4384 | — | — | — | — | — | — | — | — | — | — | 0.7597 | -0.0014 | 0.4691 | 0.8064 | 0.8658 | — | — | — | — | — | — | — | |
| FDriver-tiny# model params=7 M2024.12 | 1.4475 | — | — | — | — | — | — | — | — | — | — | 0.7584 | -0.003 | 0.4614 | 0.8069 | 0.8658 | — | — | — | — | — | — | — | |
| FDriver-tinyBackbone=Tiny2024.12 | 1.4475 | — | — | — | — | — | — | 96.41 | 0.8366 | — | 94.46 | — | — | — | — | — | 0.7584 | 0.3661 | 0.3669 | 0.4876 | 0.6248 | 0.384 | 0.6688 | |
| VBD#Param=12M2024.05 | 1.4743 | 0.72 | 0.3591 | 0.3664 | 0.4197 | 0.5222 | 0.3683 | 93.41 | 0.8153 | 0.6508 | 87.88 | — | — | — | — | — | — | — | — | — | — | — | — | |
| SMART# model params=8 M2024.12 | 1.5447 | — | — | — | — | — | — | — | — | — | — | 0.7511 | -0.0103 | 0.4445 | 0.805 | 0.8571 | — | — | — | — | — | — | — | |
| SMART2024.12 | 1.5447 | — | — | — | — | — | — | 96.55 | 0.8318 | — | 93.63 | — | — | — | — | — | 0.7511 | 0.3646 | 0.4057 | 0.4231 | 0.5845 | 0.3769 | 0.659 | |
| GUMP2024.12 | 1.6031 | — | — | — | — | — | — | 94.03 | 0.8276 | — | 90.28 | — | — | — | — | — | 0.7431 | 0.3569 | 0.4111 | 0.5089 | 0.6353 | 0.3707 | 0.6686 | |
| GUMP#Param=523M2024.05 | 1.6041 | 0.7431 | 0.3567 | 0.4111 | 0.5089 | 0.6353 | 0.3707 | 94.03 | 0.8276 | 0.6686 | 90.28 | — | — | — | — | — | — | — | — | — | — | — | — | |
| GUMP# model params=523 M2024.12 | 1.6041 | — | — | — | — | — | — | — | — | — | — | 0.7431 | -0.0183 | 0.478 | 0.7887 | 0.8359 | — | — | — | — | — | — | — | |
| MVTE#Param=>65M2024.05 | 1.677 | 0.7302 | 0.3506 | 0.3531 | 0.4974 | 0.6 | 0.3743 | 90.49 | 0.831 | 0.6655 | 90.71 | — | — | — | — | — | — | — | — | — | — | — | — | |
| TrafficBots V1.5#Param=10M2024.05 | 1.8825 | 0.6988 | 0.3361 | 0.3497 | 0.4512 | 0.5844 | 0.3596 | 80.83 | 0.8209 | 0.6423 | 91.34 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Linear Extrapolation2024.05 | 7.5148 | 0.3985 | 0.0434 | 0.1661 | 0.2522 | 0.4393 | 0.2154 | 39.05 | 0.7555 | 0.4801 | 44.26 | — | — | — | — | — | — | — | — | — | — | — | — | |
| BehaviorGPTType=Regression-Based, Anchor-Based Matching=false, Continuous Regression=true, Prediction Horizon=1s, Number of Components=16, Closed-Loop Samples=false2025.01 | — | 0.7473 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GUMPType=Discrete NTP, Anchor-Based Matching=true, Continuous Regression=false, Prediction Horizon=2s, Number of Components=1.6 × 10^33†, Closed-Loop Samples=true2025.01 | — | 0.7431 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| KiGRASType=Discrete NTP, Anchor-Based Matching=true, Continuous Regression=false, Prediction Horizon=0.5s, Number of Components=3969, Closed-Loop Samples=unclear*2025.01 | — | 0.7597 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MVTEType=Regression-Based, Anchor-Based Matching=true, Continuous Regression=true, Prediction Horizon=1s, Number of Components=64, Closed-Loop Samples=false2025.01 | — | 0.7302 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SMARTType=Discrete NTP, Anchor-Based Matching=true, Continuous Regression=false, Prediction Horizon=0.5s, Number of Components=1024, Closed-Loop Samples=true2025.01 | — | 0.7591 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |