Multimodal AutoML on Multimodal Benchmark
91.3Prod ScoreELECTRA + Exponential Decay
Evaluation Results
| Method | Links | ||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ELECTRA + Exponential DecayNLP Backbone=ELECTRA, Fine-tuning Tricks=Exponential Decay r = 0.82021.11 | 91.3 | 44.1 | 41.8 | 74.5 | 33.7 | 95.3 | 57.9 | 96.3 | 85.2 | 96.3 | 76 | 66.4 | 80.8 | 0.4 | 44.7 | 56.8 | 84.1 | 63.2 | 69.1 | 0.12 | |
| Text-EmbeddingAggregation Strategy=Text-Embedding2021.11 | 90.9 | 45.6 | 43.8 | 75.1 | 43 | 97.2 | 43.4 | 58.7 | 85.5 | 96.2 | 79 | 65.8 | 83 | 0.8 | 43.8 | 59.4 | 93.2 | 63.8 | 68.2 | 0.17 | |
| ELECTRANLP Backbone=ELECTRA2021.11 | 90.7 | 45.4 | 41.9 | 74.6 | 34.9 | 95.5 | 58.6 | 96.5 | 75 | 82.4 | 75.4 | 60.6 | 81.3 | 0.3 | 45.7 | 46.6 | 85.7 | 60.5 | 66.6 | 0.09 | |
| ELECTRA + Exponential Decay + Average 3NLP Backbone=ELECTRA, Fine-tuning Tricks=Exponential Decay r = 0.8, Average 32021.11 | 90.7 | 44.9 | 44.5 | 74.7 | 35 | 95.4 | 58.1 | 96.5 | 85.8 | 96.1 | 76.6 | 65.6 | 80.7 | 0.4 | 44.5 | 57.1 | 84.1 | 63.5 | 69.4 | 0.14 | |
| Pre-EmbeddingAggregation Strategy=Pre-Embedding2021.11 | 90.7 | 43.9 | 42.9 | 74.4 | 44.9 | 97.2 | 43.3 | 58.6 | 87.1 | 92.6 | 74.3 | 49.1 | 68 | 1.2 | 46 | 58.1 | 93.9 | 59.3 | 68.2 | 0.11 | |
| Stack-EnsembleAggregation Strategy=Stack-Ensemble2021.11 | 90.7 | 43.9 | 43.7 | 74.9 | 45.9 | 97.7 | 60.5 | 96.7 | 87.8 | 96.4 | 79.7 | 62.4 | 83.6 | 2 | 49.6 | 63.8 | 94.3 | 68.4 | 72.6 | 0.69 | |
| ROBERTaNLP Backbone=ROBERTa2021.11 | 89.8 | 41.2 | 26.8 | 70 | 34.4 | 95.3 | 56.1 | 96 | 73.1 | 92.9 | 75.1 | 61.5 | 81.1 | 0 | 39.6 | 15.1 | 82.1 | 57.2 | 61.1 | 0.07 | |
| Multimodal-EmbeddingAggregation Strategy=Multimodal-Embedding2021.11 | 89.5 | 21.6 | 24.7 | 64.2 | 43.8 | 97.4 | 43.2 | 58.7 | 84.7 | 96.7 | 79.4 | 68.3 | 82.9 | 0.7 | 45.1 | 59.5 | 93.4 | 64.4 | 65.7 | 0.25 | |
| AG-Stack + N-GramAutoML Framework=AutoGluon, Aggregation=Stack, Feature Engineering=N-Gram2021.11 | 89.5 | 41.4 | 38.3 | 62.4 | 46.6 | 97.9 | 56.9 | 91.5 | 85 | 96.8 | 77.5 | 61.2 | 84.2 | 2 | 49.4 | 60 | 94.3 | 66.3 | 71.2 | 0.43 | |
| AG-Weighted + N-GramAutoML Framework=AutoGluon, Aggregation=Weighted, Feature Engineering=N-Gram2021.11 | 89.2 | 42.6 | 38.2 | 61 | 45 | 97.8 | 52.6 | 90.9 | 84.2 | 96.6 | 77.2 | 35.7 | 82.9 | 1.9 | 48.4 | 59.1 | 94.1 | 64 | 69.4 | 0.17 | |
| AG-WeightedAutoML Framework=AutoGluon, Aggregation=Weighted2021.11 | 89.1 | 4.6 | 7.6 | -0.2 | 42.6 | 84.1 | 9.8 | 58.7 | 84.5 | 68.6 | 66.8 | 0.4 | 17.3 | 1.6 | 22.6 | 22.2 | 93.4 | 40.5 | 40.2 | 0.08 | |
| AG-StackAutoML Framework=AutoGluon, Aggregation=Stack2021.11 | 89.1 | 4.6 | 7.7 | 0.1 | 43.5 | 84.1 | 9.8 | 58.7 | 84.4 | 69.7 | 67 | 0.3 | 17.5 | 1.7 | 22.6 | 23.3 | 93.4 | 40.7 | 40.5 | 0.09 | |
| H2O AutoMLAutoML Framework=H2O AutoML2021.11 | 86.9 | 24.7 | 15.9 | 16.3 | 32.9 | 97.6 | 43 | 53.1 | 81.3 | 75.6 | 66.9 | 41.1 | 47.8 | 1.4 | 52.5 | 44.4 | 93.9 | 51.6 | 51.8 | 0.11 | |
| Weighted-EnsembleAggregation Strategy=Weighted-Ensemble2021.11 | 86.7 | 44.6 | 43.2 | 74.8 | 45.3 | 97.6 | 59.7 | 95.7 | 87.6 | 92.3 | 78.7 | 64.1 | 81.4 | 1.8 | 48.3 | 62 | 94.1 | 67.6 | 71.7 | 0.25 | |
| H2O AutoML + Word2VecAutoML Framework=H2O AutoML, Feature Engineering=Word2Vec2021.11 | 85.9 | 24.4 | 28.5 | 54.6 | 34.7 | 97.3 | 53.4 | 84.7 | 82.7 | 94.3 | 75.5 | 44.3 | 77.8 | 1.3 | 52.8 | 58.6 | 93.2 | 61.3 | 62.8 | 0.14 | |
| H2O AutoML + Pre-EmbeddingAutoML Framework=H2O AutoML, Feature Engineering=Pre-Embedding2021.11 | 84.6 | 22.7 | 31.2 | 64.4 | 36.7 | 96.9 | 28.2 | 52.6 | 87.4 | 89.3 | 73.8 | 54.9 | 57.1 | 0.7 | 48.3 | 48.3 | 52.3 | 93.3 | 57.2 | 0.09 | |
| Fuse-LateFusion Strategy=Fuse-Late, Note=Identified as Multimodal-Net2021.11 | 72.9 | 45.1 | 43.2 | 74.6 | 39.5 | 95.8 | 60.3 | 96.6 | 85.7 | 96.1 | 77.3 | 63.9 | 81.2 | 1.5 | 46.8 | 57.1 | 90.7 | 66.4 | 66.8 | 0.22 | |
| Fuse-EarlyFusion Strategy=Fuse-Early2021.11 | 72.8 | 43.6 | 43.1 | 74.3 | 37.7 | 95.3 | 59.6 | 96.7 | 84.3 | 96 | 77 | 65.3 | 80.6 | 1.3 | 45.8 | 54.8 | 90.1 | 65.8 | 65.9 | 0.21 | |
| All-TextFusion Strategy=All-Text2021.11 | 70.5 | 41 | 35.6 | 71.8 | 36.6 | 95.7 | 59.9 | 96.7 | 84 | 96.7 | 79.9 | 64.5 | 81 | 1.3 | 46.5 | 58.5 | 89.2 | 66.2 | 65.4 | 0.28 |