Multimodal Sentiment Analysis on MOSEI
0.486MAEHyperEmo-LLaMA2-7B
Evaluation results
| Method | Links | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| HyperEmo-LLaMA2-7BBackbone=LLaMA2-7B2026.05 | 0.486 | 88.69 | 57.98 | 88.74 | — | 0.806 | — | — | — | — | — | — | — | — | — | |
| HyperEmo-ChatGLM3-6BBackbone=ChatGLM3-6B2026.05 | 0.494 | 89.06 | 57.32 | 88.82 | — | 0.801 | — | — | — | — | — | — | — | — | — | |
| EGMFBackbone=GLM3-6B2026.01 | 0.496 | 87.3 | 55.38 | 87.09 | — | 0.801 | — | — | — | — | — | — | — | — | — | |
| GRAMformer2026.06 | 0.4963 | — | 56.05 | — | — | 0.81 | 58.07 | — | — | 84.07 | 87.39 | 84.33 | 87.26 | — | — | |
| GRAMformerEncoding Setting=RoBERTa-Large + Data2Vec (Video features pre-extracted), Params=731.6M2026.06 | 0.4963 | 84.07 | 56.05 | 84.33 | — | 0.0081 | 58.07 | — | — | — | — | — | — | 87.39 | 87.26 | |
| EGMFBackbone=llama2-7B2026.01 | 0.5 | 87.16 | 54.73 | 86.97 | — | 0.796 | — | — | — | — | — | — | — | — | — | |
| EGMFBackbone=GLM4-9B2026.01 | 0.514 | 87.08 | 54.78 | 87 | — | 0.79 | — | — | — | — | — | — | — | — | — | |
| MMML2026.06 | 0.517 | — | 54.95 | — | — | 0.79 | 57.32 | — | — | 86.32 | 86.73 | 86.23 | 86.49 | — | — | |
| FDMA2026.06 | 0.517 | — | 55.3 | — | — | 0.782 | — | — | — | — | 87.41 | — | 87.38 | — | — | |
| MMMLEncoding Setting=RoBERTa-Large + Data2Vec (Video features pre-extracted), Params=889.6M2026.06 | 0.517 | 86.32 | 54.95 | 86.23 | — | 0.0079 | 57.32 | — | — | — | — | — | — | 86.73 | 86.49 | |
| ConFEDE2026.06 | 0.522 | — | 54.86 | — | — | 0.78 | — | — | — | 81.65 | 85.82 | 82.17 | 85.83 | — | — | |
| UniMSE2026.01 | 0.523 | 85.86 | 54.39 | 85.79 | — | 0.773 | — | — | — | — | — | — | — | — | — | |
| UniMSE2026.05 | 0.523 | 85.86 | 54.39 | 85.79 | — | 0.773 | — | — | — | — | — | — | — | — | — | |
| CHFN2026.01 | 0.525 | 83.7 | 54.3 | 83.9 | — | 0.778 | — | — | — | — | — | — | — | — | — | |
| CHFN2026.05 | 0.525 | 83.7 | 54.3 | 83.9 | — | 0.778 | — | — | — | — | — | — | — | — | — | |
| MMIM2026.05 | 0.526 | 82.24 | 54.24 | 82.66 | — | 0.772 | — | — | — | — | — | — | — | — | — | |
| Self-MMEncoding Setting=BERT Finetuning, Params=109.6M2026.06 | 0.5287 | 80.19 | 54.14 | 80.75 | — | 0.7643 | 55.91 | — | — | — | — | — | — | 84.16 | 84.14 | |
| Self-MM2026.01 | 0.53 | 82.81 | 53.46 | 82.53 | — | 0.765 | — | — | — | — | — | — | — | — | — | |
| Self-MM2026.05 | 0.53 | 82.81 | 53.46 | 82.53 | — | 0.765 | — | — | — | — | — | — | — | — | — | |
| GLoMo + MS-Mix (ours)Model Architecture=GLoMo, Mixup Strategy=MS-Mix2025.10 | 0.531 | 85.32 | 53.15 | 85.25 | — | — | 55.84 | 74.64 | — | — | — | — | — | — | — | |
| GLoMo + P MixModel Architecture=GLoMo, Mixup Strategy=P Mix2025.10 | 0.533 | 85.08 | 53.07 | 85.12 | — | — | 55.08 | 74.15 | — | — | — | — | — | — | — | |
| HyperEmo-Qwen-1.8BBackbone=Qwen-1.8B2026.05 | 0.533 | 86.43 | 53.87 | 86.19 | — | 0.756 | — | — | — | — | — | — | — | — | — | |
| ALMTreproduced=true2026.06 | 0.534 | — | 53.32 | — | — | 0.771 | 55.05 | — | — | 82.34 | 84.95 | 81.85 | 85.93 | — | — | |
| GLoMo + Manifold MixModel Architecture=GLoMo, Mixup Strategy=Manifold Mix2025.10 | 0.535 | 84.88 | 53.37 | 85.13 | — | — | 56.08 | 74.4 | — | — | — | — | — | — | — | |
| GLoMo + MultiMixModel Architecture=GLoMo, Mixup Strategy=MultiMix2025.10 | 0.535 | 85.13 | 52.93 | 85.25 | — | — | 55.05 | 74.09 | — | — | — | — | — | — | — | |
| Self-MMreproduced=true2026.06 | 0.535 | — | 53.65 | — | — | 0.761 | 53.54 | — | — | 82.09 | 84.76 | 82.43 | 84.67 | — | — | |
| ALMT + MS-Mix (ours)Model Architecture=ALMT, Mixup Strategy=MS-Mix2025.10 | 0.536 | 85.41 | 53.55 | 85.62 | — | — | 55.17 | 74.29 | — | — | — | — | — | — | — | |
| FDMER2026.06 | 0.536 | — | 54.1 | — | — | 0.773 | — | — | — | — | 86.1 | — | 85.8 | — | — | |
| GLoMo + AdvMixUpModel Architecture=GLoMo, Mixup Strategy=AdvMixUp2025.10 | 0.537 | 85.24 | 53.04 | 84.92 | — | — | 55 | 74.06 | — | — | — | — | — | — | — | |
| ALMT + MultiMixModel Architecture=ALMT, Mixup Strategy=MultiMix2025.10 | 0.538 | 85.1 | 52.7 | 85.22 | — | — | 54.7 | 73.61 | — | — | — | — | — | — | — | |
| ALMT + P MixModel Architecture=ALMT, Mixup Strategy=P Mix2025.10 | 0.539 | 85.55 | 53.13 | 85.48 | — | — | 54.58 | 73.91 | — | — | — | — | — | — | — | |
| DEVA2026.06 | 0.541 | — | 52.26 | — | — | 0.769 | 55.32 | — | — | 83.26 | 86.13 | 82.93 | 86.21 | — | — | |
| TeTFNEncoding Setting=BERT Finetuning, Params=110.7M2026.06 | 0.5413 | 81.44 | 53.9 | 81.92 | — | 0.7612 | 55.82 | — | — | — | — | — | — | 85.15 | 85.11 | |
| MISA + MS-Mix (ours)Model Architecture=MISA, Mixup Strategy=MS-Mix2025.10 | 0.542 | 84.84 | 52.95 | 84.72 | — | — | 55.04 | 73.96 | — | — | — | — | — | — | — | |
| GLoMoModel Architecture=GLoMo, Mixup Strategy=None2025.10 | 0.542 | 84.93 | 52.67 | 85.28 | — | — | 53.8 | 73.72 | — | — | — | — | — | — | — | |
| ALMT + Manifold MixModel Architecture=ALMT, Mixup Strategy=Manifold Mix2025.10 | 0.543 | 85.13 | 52.65 | 85.08 | — | — | 54.33 | 73.33 | — | — | — | — | — | — | — | |
| MAG-BERT2026.06 | 0.543 | — | 52.67 | — | — | 0.755 | — | — | — | 82.51 | 84.82 | 82.77 | 84.71 | — | — | |
| MISA + AdvMixUpModel Architecture=MISA, Mixup Strategy=AdvMixUp2025.10 | 0.544 | 84.65 | 52.69 | 84.55 | — | — | 54.79 | 73.34 | — | — | — | — | — | — | — | |
| ALMT + AdvMixUpModel Architecture=ALMT, Mixup Strategy=AdvMixUp2025.10 | 0.544 | 85.22 | 52.87 | 85.05 | — | — | 54.46 | 73.6 | — | — | — | — | — | — | — | |
| UniSAT52026.01 | 0.546 | 84.22 | 52.5 | — | — | — | — | — | — | — | — | — | — | — | — | |
| UniSAT52026.05 | 0.546 | 84.22 | 52.5 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MISA + P MixModel Architecture=MISA, Mixup Strategy=P Mix2025.10 | 0.547 | 84.49 | 52.2 | 84.19 | — | — | 54.79 | 73.27 | — | — | — | — | — | — | — | |
| GRAMformerEncoding Setting=BERT Finetuning, Params=110.0M2026.06 | 0.5479 | 80.97 | 53.02 | 81.48 | — | 0.761 | 54.78 | — | — | — | — | — | — | 84.89 | 84.85 | |
| MISA + Manifold MixModel Architecture=MISA, Mixup Strategy=Manifold Mix2025.10 | 0.548 | 84.51 | 52.83 | 84.57 | — | — | 54.49 | 72.9 | — | — | — | — | — | — | — | |
| ALMTModel Architecture=ALMT, Mixup Strategy=None2025.10 | 0.55 | 85.15 | 52.73 | 85.04 | — | — | 54.11 | 73.32 | — | — | — | — | — | — | — | |
| MISA + MultiMixModel Architecture=MISA, Mixup Strategy=MultiMix2025.10 | 0.552 | 84.23 | 52.66 | 84.2 | — | — | 54.07 | 72.95 | — | — | — | — | — | — | — | |
| MuIT + MS-Mix (ours)Model Architecture=MuIT, Mixup Strategy=MS-Mix2025.10 | 0.555 | 84.98 | 53.04 | 84.87 | — | — | 55 | 73.69 | — | — | — | — | — | — | — | |
| MISA2026.05 | 0.555 | 83.6 | 52.2 | 83.8 | — | 0.756 | — | — | — | — | — | — | — | — | — | |
| LMF + MS-Mix (ours)Model Architecture=LMF, Mixup Strategy=MS-Mix2025.10 | 0.556 | 84.53 | 52.65 | 84.6 | — | — | 54.45 | 73.71 | — | — | — | — | — | — | — | |
| MISAsource=obtained from [30]2026.06 | 0.557 | — | 52.05 | — | — | 0.751 | 53.63 | — | — | 80.67 | 84.67 | 81.12 | 84.66 | — | — | |
| MISAModel Architecture=MISA, Mixup Strategy=None2025.10 | 0.558 | 84.67 | 52.05 | 84.66 | — | — | 53.63 | 72.8 | — | — | — | — | — | — | — | |
| MulT2026.01 | 0.559 | 81.15 | 52.84 | 81.56 | — | 0.733 | — | — | — | — | — | — | — | — | — | |
| TFN + MS-Mix (ours)Model Architecture=TFN, Mixup Strategy=MS-Mix2025.10 | 0.559 | 83.41 | 52.2 | 83.3 | — | — | 53.87 | 72.84 | — | — | — | — | — | — | — | |
| MuITModel Architecture=MuIT, Mixup Strategy=None2025.10 | 0.559 | 84.63 | 52.84 | 84.52 | — | — | 54.51 | 73.2 | — | — | — | — | — | — | — | |
| MuIT + P MixModel Architecture=MuIT, Mixup Strategy=P Mix2025.10 | 0.559 | 84.71 | 52.68 | 84.65 | — | — | 54.37 | 73.33 | — | — | — | — | — | — | — | |
| MulT2026.05 | 0.559 | 81.15 | 52.84 | 81.56 | — | 0.733 | — | — | — | — | — | — | — | — | — | |
| MulTsource=obtained from [30]2026.06 | 0.559 | — | 52.84 | — | — | 0.733 | 54.18 | — | — | 81.15 | 84.63 | 81.56 | 84.52 | — | — | |
| LMF + P MixModel Architecture=LMF, Mixup Strategy=P Mix2025.10 | 0.56 | 83.95 | 52.73 | 84.02 | — | — | 54.37 | 72.88 | — | — | — | — | — | — | — | |
| LMFEncoding Setting=Pre-extracted Features (BERT text encoder), Params=0.51M2026.06 | 0.5607 | 81.3 | 52.57 | 81.67 | — | 0.7351 | 53.96 | — | — | — | — | — | — | 84.28 | 84.16 | |
| MuIT + Manifold MixModel Architecture=MuIT, Mixup Strategy=Manifold Mix2025.10 | 0.562 | 84.39 | 52.39 | 84.32 | — | — | 53.94 | 72.98 | — | — | — | — | — | — | — | |
| LF-DNNEncoding Setting=Pre-extracted Features (BERT text encoder), Params=0.57M2026.06 | 0.563 | 78.07 | 52.26 | 78.57 | — | 0.7317 | 53.54 | — | — | — | — | — | — | 82.15 | 82 | |
| MULTEncoding Setting=Pre-extracted Features (BERT text encoder), Params=0.98M2026.06 | 0.5637 | 80.52 | 52.59 | 80.94 | — | 0.7303 | 54.04 | — | — | — | — | — | — | 83.77 | 83.65 | |
| TFN + P MixModel Architecture=TFN, Mixup Strategy=P Mix2025.10 | 0.564 | 83.36 | 51.64 | 83.37 | — | — | 53.08 | 72.37 | — | — | — | — | — | — | — | |
| LMF + MultiMixModel Architecture=LMF, Mixup Strategy=MultiMix2025.10 | 0.564 | 84.17 | 52.72 | 84.29 | — | — | 53.72 | 72.74 | — | — | — | — | — | — | — | |
| LMF + AdvMixUpModel Architecture=LMF, Mixup Strategy=AdvMixUp2025.10 | 0.565 | 84.12 | 52.14 | 84.12 | — | — | 53.47 | 72.9 | — | — | — | — | — | — | — | |
| MuIT + MultiMixModel Architecture=MuIT, Mixup Strategy=MultiMix2025.10 | 0.565 | 83.99 | 52.46 | 84.03 | — | — | 54.22 | 72.98 | — | — | — | — | — | — | — | |
| ICCN2026.06 | 0.565 | — | 51.58 | — | — | 0.713 | — | — | — | — | 84.18 | — | 84.15 | — | — | |
| TFN + AdvMixUpModel Architecture=TFN, Mixup Strategy=AdvMixUp2025.10 | 0.566 | 82.85 | 52.2 | 82.96 | — | — | 53.32 | 72.18 | — | — | — | — | — | — | — | |
| Graph-MFNEncoding Setting=Pre-extracted Features (BERT text encoder), Params=0.68M2026.06 | 0.5664 | 82.05 | 51.86 | 82.25 | — | 0.7261 | 53.17 | — | — | — | — | — | — | 84.02 | 83.81 | |
| TFN + MultiMixModel Architecture=TFN, Mixup Strategy=MultiMix2025.10 | 0.569 | 82.73 | 51.74 | 82.46 | — | — | 53.17 | 71.86 | — | — | — | — | — | — | — | |
| MuIT + AdvMixUpModel Architecture=MuIT, Mixup Strategy=AdvMixUp2025.10 | 0.57 | 84.31 | 51.98 | 84.29 | — | — | 54.3 | 72.83 | — | — | — | — | — | — | — | |
| MFNEncoding Setting=Pre-extracted Features (BERT text encoder), Params=127.7M2026.06 | 0.5706 | 80.2 | 51.4 | 80.7 | — | 0.7187 | 52.8 | — | — | — | — | — | — | 83.59 | 83.54 | |
| TFN + Manifold MixModel Architecture=TFN, Mixup Strategy=Manifold Mix2025.10 | 0.571 | 83.41 | 51.25 | 83.42 | — | — | 52.87 | 72.31 | — | — | — | — | — | — | — | |
| LMF + Manifold MixModel Architecture=LMF, Mixup Strategy=Manifold Mix2025.10 | 0.572 | 83.21 | 52.04 | 83.39 | — | — | 53.4 | 71.66 | — | — | — | — | — | — | — | |
| TFNsource=obtained from [30]2026.06 | 0.572 | — | 51.6 | — | — | 0.714 | — | — | — | 78.5 | 81.89 | 78.96 | 81.74 | — | — | |
| TFNEncoding Setting=Pre-extracted Features (BERT text encoder), Params=5.04M2026.06 | 0.5722 | 76.58 | 51.81 | 77.25 | — | 0.7195 | 53.25 | — | — | — | — | — | — | 81.47 | 81.41 | |
| TFNModel Architecture=TFN, Mixup Strategy=None2025.10 | 0.573 | 81.89 | 51.6 | 81.74 | — | — | 53.1 | 70.97 | — | — | — | — | — | — | — | |
| TFN2026.05 | 0.573 | 78.5 | 51.6 | 78.96 | — | 0.714 | — | — | — | — | — | — | — | — | — | |
| MFSA2026.06 | 0.574 | — | 53.2 | — | — | 0.724 | — | — | — | — | 83.8 | — | 83.6 | — | — | |
| LMFsource=obtained from [30]2026.06 | 0.575 | — | 51.59 | — | — | 0.716 | — | — | — | 80.54 | 83.48 | 80.94 | 83.36 | — | — | |
| LMFModel Architecture=LMF, Mixup Strategy=None2025.10 | 0.576 | 83.48 | 51.59 | 83.36 | — | — | 52.99 | 72.15 | — | — | — | — | — | — | — | |
| LMF2026.05 | 0.576 | 80.54 | 51.59 | 80.94 | — | 0.717 | — | — | — | — | — | — | — | — | — | |
| GRAMformerEncoding Setting=Pre-extracted Features (BERT text encoder), Params=0.52M2026.06 | 0.5822 | 81.49 | 51.42 | 81.71 | — | 0.7084 | 52.78 | — | — | — | — | — | — | 83.55 | 83.33 | |
| MAG-BERT2026.01 | 0.583 | 82.51 | 50.41 | 82.77 | — | 0.741 | — | — | — | — | — | — | — | — | — | |
| MAG-BERT2026.05 | 0.583 | 82.51 | 50.41 | 82.77 | — | 0.741 | — | — | — | — | — | — | — | — | — | |
| UniSABART2026.01 | 0.587 | 84.93 | 50.03 | — | — | — | — | — | — | — | — | — | — | — | — | |
| UniSABART2026.05 | 0.587 | 84.93 | 50.03 | — | — | — | — | — | — | — | — | — | — | — | — | |
| FedUAFtraining_missing_modality_ratio=0.2, data_distribution=IID, test_missing_modality_ratio=0.22026.02 | 0.63 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MIFLtraining_missing_modality_ratio=0.2, data_distribution=IID, test_missing_modality_ratio=0.42026.02 | 0.64 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FedUAFtraining_missing_modality_ratio=0.2, data_distribution=IID, test_missing_modality_ratio=0.42026.02 | 0.65 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FedProxtraining_missing_modality_ratio=0.2, data_distribution=IID, test_missing_modality_ratio=0.22026.02 | 0.66 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| EGMFBackbone=llama3-8B2026.01 | 0.67 | 86.75 | 47.83 | 86.58 | — | 0.713 | — | — | — | — | — | — | — | — | — | |
| FedAvgtraining_missing_modality_ratio=0.2, data_distribution=IID, test_missing_modality_ratio=0.42026.02 | 0.67 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MIFLtraining_missing_modality_ratio=0.2, data_distribution=IID, test_missing_modality_ratio=0.22026.02 | 0.67 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FedUAFtraining_missing_modality_ratio=0.2, data_distribution=IID, test_missing_modality_ratio=0.62026.02 | 0.67 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FedMSplittraining_missing_modality_ratio=0.2, data_distribution=IID, test_missing_modality_ratio=0.22026.02 | 0.68 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FedMACtraining_missing_modality_ratio=0.2, data_distribution=IID, test_missing_modality_ratio=0.42026.02 | 0.68 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FedUAFtraining_missing_modality_ratio=0.2, data_distribution=Non-IID, test_missing_modality_ratio=0.22026.02 | 0.68 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FedMACtraining_missing_modality_ratio=0.2, data_distribution=IID, test_missing_modality_ratio=0.22026.02 | 0.69 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FedUAFtraining_missing_modality_ratio=0.2, data_distribution=IID, test_missing_modality_ratio=0.82026.02 | 0.69 | — | — | — | — | — | — | — | — | — | — | — | — | — | — |