Meme Intervention Generation on ICMM (test)
24.42ROUGE-1MemeGuard
Evaluation results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| MemeGuardBase LLM=GPT3.5-Turbo, Input=OCR + Filtered Knowledge2024.06 | 24.42 | 4.7 | 15.22 | 39.45 | 32.32 | 17.82 | 11.91 | 25.37 | 19.03 | 89.02 | |
| OCR + VLmemeBase LLM=GPT3.5-Turbo, Input=OCR + VLmeme knowledge2024.06 | 22.44 | 3.13 | 10.18 | 37.76 | 30.23 | 17.53 | 11.35 | 24.22 | 14.33 | 86.26 | |
| OCRBase LLM=GPT3.5-Turbo, Input=OCR text2024.06 | 21.05 | 3.85 | 13.25 | 37.9 | 29.57 | 17.53 | 11.24 | 24.06 | 17.08 | 85.07 | |
| MemeGuardBase LLM=FLAN-T5, Input=OCR + Filtered Knowledge2024.06 | 14.11 | 2.01 | 11.22 | 48.91 | 30.21 | 12.81 | 6.6 | 24.63 | 15.41 | 87.21 | |
| OCR + VLmemeBase LLM=FLAN-T5, Input=OCR + VLmeme knowledge2024.06 | 13.96 | 0.77 | 10.91 | 45.47 | 27.98 | 12.33 | 7.22 | 23.25 | 14.85 | 85.04 | |
| OCR + MiniGPT4Base LLM=FLAN-T5, Input=OCR + MiniGPT4 description2024.06 | 13.88 | 1.03 | 10.13 | 39.65 | 26.68 | 12.75 | 6.58 | 21.41 | 13.75 | 84.5 | |
| OCR + MiniGPT4Base LLM=GPT3.5-Turbo, Input=OCR + MiniGPT4 description2024.06 | 13.71 | 1.83 | 9.3 | 21.3 | 17.99 | 10.97 | 6.51 | 14.19 | 11.23 | 85.33 | |
| OCR + MiniGPT4Base LLM=RedPajama, Input=OCR + MiniGPT4 description2024.06 | 11.61 | 0.7 | 8.55 | 33.94 | 22.19 | 10.6 | 5.2 | 17.98 | 11.58 | 82.4 | |
| MemeGuardBase LLM=RedPajama, Input=OCR + Filtered Knowledge2024.06 | 10.83 | 0.57 | 9.01 | 35.23 | 22.82 | 11.05 | 4.9 | 18.5 | 12.11 | 83.42 | |
| OCR + VLmemeBase LLM=RedPajama, Input=OCR + VLmeme knowledge2024.06 | 10.82 | 0.58 | 7.93 | 35.1 | 22.72 | 10.02 | 4.88 | 18.18 | 11.04 | 80.93 | |
| MemeGuardBase LLM=LLaMA, Input=OCR + Filtered Knowledge2024.06 | 9.27 | 1.71 | 5.29 | 31.96 | 29.08 | 11.89 | 4.3 | 19.31 | 8.3 | 80.11 | |
| OCR + MiniGPT4Base LLM=LLaMA, Input=OCR + MiniGPT4 description2024.06 | 8.84 | 0.87 | 6.98 | 31.19 | 27.77 | 10.64 | 4.74 | 18.58 | 10.15 | 79.98 | |
| MemeGuardBase LLM=Dolly, Input=OCR + Filtered Knowledge2024.06 | 8.68 | 0.41 | 7.22 | 17.05 | 12.88 | 6.65 | 2.86 | 9.86 | 8.33 | 82.27 | |
| OCR + VLmemeBase LLM=Dolly, Input=OCR + VLmeme knowledge2024.06 | 8.48 | 0.23 | 6.26 | 16.55 | 12.11 | 6.17 | 3.36 | 9.55 | 7.56 | 79.17 | |
| OCRBase LLM=LLaMA, Input=OCR text2024.06 | 8.48 | 0.85 | 6.96 | 31.15 | 27.7 | 10.56 | 4.6 | 18.5 | 10.11 | 79.98 | |
| OCRBase LLM=RedPajama, Input=OCR text2024.06 | 8.31 | 0.45 | 6.63 | 39.17 | 19.9 | 7.47 | 3.34 | 17.47 | 9.61 | 79.56 | |
| OCR + VLmemeBase LLM=LLaMA, Input=OCR + VLmeme knowledge2024.06 | 6.18 | 0.61 | 4.73 | 30.89 | 28.4 | 10.95 | 4.78 | 18.75 | 7.55 | 80.11 | |
| OCRBase LLM=FLAN-T5, Input=OCR text2024.06 | 5.92 | 0.4 | 4.96 | 10.78 | 4.16 | 1.34 | 0.59 | 4.22 | 4.55 | 83.19 | |
| OCRBase LLM=Dolly, Input=OCR text2024.06 | 2.39 | 0.15 | 2.07 | 7.49 | 4.52 | 1.84 | 0.84 | 3.67 | 2.65 | 74.35 | |
| OCR + MiniGPT4Base LLM=Dolly, Input=OCR + MiniGPT4 description2024.06 | 0.94 | 0.05 | 0.74 | 2.77 | 1.73 | 0.75 | 0.36 | 1.4 | 0.97 | 73.81 |