Bot detection on TwiBot-20
89.9AccuracyChatGPT
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| ChatGPTModality=ENSEMBLE, Instruction Tuning=true2024.02 | 89.9 | 91.5 | — | 86.1 | 97.6 | |
| ChatGPTModality=STRUCT-RAND, Instruction Tuning=true2024.02 | 89 | 90.4 | — | 83.9 | 98 | |
| ChatGPTModality=STRUCT-ATT, Instruction Tuning=true2024.02 | 88.5 | 88.8 | — | 85.6 | 92.3 | |
| BICText Modality=true, Graph Modality=true, Modality Interaction=true2022.08 | 87.61 | 89.13 | — | — | — | |
| BICMetadata (M)=true, Texts (T)=true, Neighborhoods (N)=false2024.02 | 87.6 | 89.1 | — | — | — | |
| RGTMetadata (M)=true, Texts (T)=true, Neighborhoods (N)=true2024.02 | 86.6 | 88 | — | 85.2 | 91.1 | |
| RGTText Modality=true, Graph Modality=true, Modality Interaction=false2022.08 | 86.57 | 88.01 | — | — | — | |
| BIC w/o TextText Modality=false, Graph Modality=true, Modality Interaction=false2022.08 | 85.78 | 87.25 | — | — | — | |
| LMBOTMetadata (M)=true, Texts (T)=true, Neighborhoods (N)=true2024.02 | 85.6 | 87.6 | — | — | — | |
| BIC w/o GraphText Modality=true, Graph Modality=false, Modality Interaction=false2022.08 | 85.44 | 86.97 | — | — | — | |
| CACLBackbone=HGT2024.05 | 85.12 | 87.28 | 70.75 | — | — | |
| BOTPERCENTMetadata (M)=true, Texts (T)=true, Neighborhoods (N)=true2024.02 | 84.5 | 86.5 | — | — | — | |
| BotRGCN2024.05 | 84.43 | 85.79 | 67.17 | — | — | |
| RGTClassifier=XGBoost2025.12 | 84.4 | 79.7 | — | 72 | 89.2 | |
| SATARText Modality=true, Graph Modality=true, Modality Interaction=true2022.08 | 84.02 | 86.07 | — | — | — | |
| RGT2024.05 | 84.02 | 86.07 | 68.08 | — | — | |
| SATAR2024.05 | 84.02 | 85.74 | 67.83 | — | — | |
| HGT2024.05 | 84.02 | 86.35 | 68.48 | — | — | |
| SimpleHGN2024.05 | 83.93 | 86.33 | 68.4 | — | — | |
| CACLBackbone=SAGE2024.05 | 83.6 | 86.53 | 68.95 | — | — | |
| CACLBackbone=GAT2024.05 | 83.52 | 86.39 | 68.56 | — | — | |
| BotRGCNText Modality=false, Graph Modality=true, Modality Interaction=false2022.08 | 83.27 | 85.26 | — | — | — | |
| SAGE2024.05 | 83 | 85.48 | 66.38 | — | — | |
| Heterogeneous motifs (All)Classifier=XGBoost, motif_type=Heterogeneous2025.12 | 82.7 | 86.3 | — | 81.2 | 92.1 | |
| DeeProBotClassifier=XGBoost2025.12 | 82.6 | 75.1 | — | 77.1 | 73.3 | |
| GAT2024.05 | 82.24 | 84.93 | 64.95 | — | — | |
| T5Classifier=XGBoost2025.12 | 81.9 | 79.2 | — | 72.9 | 86.7 | |
| FPClassifier=XGBoost2025.12 | 81.7 | 83.1 | — | 84 | 82.8 | |
| BotRGCNClassifier=XGBoost2025.12 | 81.7 | 83.3 | — | 82.7 | 84 | |
| Yang et al.Text Modality=false, Graph Modality=false, Modality Interaction=false2022.08 | 81.64 | 84.89 | — | — | — | |
| SGBOTMetadata (M)=false, Texts (T)=false, Neighborhoods (N)=true2024.02 | 81.6 | 84.9 | — | 76.4 | 94.9 | |
| FEDRIOalpha (α)=12026.04 | 81.48 | 80.61 | — | — | — | |
| ChatGPTModality=METADATA, Instruction Tuning=true2024.02 | 81.2 | 80.6 | — | 81.4 | 84.7 | |
| FEDRIOalpha (α)=0.52026.04 | 80.32 | 79.36 | — | — | — | |
| ARGClassifier=XGBoost2025.12 | 80.3 | 82.7 | — | 83.4 | 83 | |
| FEDACKalpha (α)=0.52026.04 | 78.26 | 76.21 | — | — | — | |
| DeeProBot2024.05 | 77.34 | 80.32 | 54.4 | — | — | |
| BOTBUSTERMetadata (M)=true, Texts (T)=true, Neighborhoods (N)=false2024.02 | 77.2 | 81.2 | — | — | — | |
| FEDACK-Aalpha (α)=12026.04 | 77.16 | 70.59 | — | — | — | |
| FEDACKalpha (α)=12026.04 | 77.08 | 71.2 | — | — | — | |
| ChatGPTModality=TEXT, Instruction Tuning=true2024.02 | 76.7 | 79.1 | — | 76.8 | 81.6 | |
| ChatGPTModality=METADATA, Instruction Tuning=false2024.02 | 76.6 | 79.3 | — | 74.2 | 85.2 | |
| LOBOMetadata (M)=true, Texts (T)=true, Neighborhoods (N)=false2024.02 | 76.2 | 80.6 | — | 74.8 | 87.8 | |
| RoBERTa2024.05 | 75.55 | 73.09 | 42.15 | — | — | |
| ROBERTAMetadata (M)=false, Texts (T)=true, Neighborhoods (N)=false2024.02 | 75.5 | 73.1 | — | 73.9 | 72.4 | |
| Heterogeneous motifs (AUC'_{upper} > 0.7)Classifier=XGBoost, motif_type=Heterogeneous, Selection criteria=AUC'_upper > 0.72025.12 | 74.9 | 80.6 | — | 74.2 | 88.2 | |
| FEDACK-Aalpha (α)=0.52026.04 | 74.7 | 73.69 | — | — | — | |
| FedProxalpha (α)=12026.04 | 74.34 | 78.46 | — | — | — | |
| FedFTGalpha (α)=12026.04 | 74.27 | 74.25 | — | — | — | |
| FEDRIOalpha (α)=0.12026.04 | 74.25 | 72.12 | — | — | — | |
| BotometerClassifier=XGBoost2025.12 | 74.2 | 78.3 | — | 79.4 | 77.3 | |
| FedGenalpha (α)=12026.04 | 74.14 | 74.69 | — | — | — | |
| FedFTGalpha (α)=0.52026.04 | 74.13 | 73.48 | — | — | — | |
| FedProxalpha (α)=0.52026.04 | 73.32 | 77.33 | — | — | — | |
| Homogeneous motifs (All)Classifier=XGBoost, motif_type=Homogeneous2025.12 | 73.3 | 78.3 | — | 75.2 | 81.8 | |
| FedGenalpha (α)=0.52026.04 | 73.12 | 73.02 | — | — | — | |
| LLAMA2-70BModality=METADATA, Instruction Tuning=false2024.02 | 72.7 | 74.1 | — | 76.2 | 72 | |
| FedDFalpha (α)=12026.04 | 72.12 | — | — | — | — | |
| FedDFalpha (α)=0.52026.04 | 71.25 | — | — | — | — | |
| Wei et al.Text Modality=true, Graph Modality=false, Modality Interaction=false2022.08 | 70.23 | 53.61 | — | — | — | |
| FEDRIOalpha (α)=0.052026.04 | 69.94 | 66.02 | — | — | — | |
| FEDACKalpha (α)=0.12026.04 | 67.81 | 65.35 | — | — | — | |
| SybilWalkClassifier=XGBoost2025.12 | 67.8 | 55.3 | — | 65.3 | 48 | |
| BGSRD2024.05 | 66.36 | 70.05 | 32.28 | — | — | |
| LLAMA2-70BModality=ENSEMBLE, Instruction Tuning=false2024.02 | 66.1 | 65.9 | — | 72.3 | 60.5 | |
| EvolveBot2024.05 | 65.83 | 69.75 | 30.79 | — | — | |
| FedDistillalpha (α)=12026.04 | 64.11 | 66.04 | — | — | — | |
| FEDACK-Aalpha (α)=0.12026.04 | 63.52 | 61.04 | — | — | — | |
| FedDistillalpha (α)=0.52026.04 | 63.34 | 63.99 | — | — | — | |
| ChatGPTModality=ENSEMBLE, Instruction Tuning=false2024.02 | 63.2 | 55.7 | — | 80.1 | 42.7 | |
| Mistral-7BModality=ENSEMBLE, Instruction Tuning=false2024.02 | 60.9 | 57.3 | — | 69.9 | 48.6 | |
| FedFTGalpha (α)=0.12026.04 | 60.14 | 60.5 | — | — | — | |
| FEDACKalpha (α)=0.052026.04 | 60.14 | 56.07 | — | — | — | |
| Alhossini et al.Text Modality=false, Graph Modality=true, Modality Interaction=false2022.08 | 59.92 | 72.09 | — | — | — | |
| Kudugunta et al.Text Modality=true, Graph Modality=false, Modality Interaction=false2022.08 | 59.59 | 47.26 | — | — | — | |
| FedGenalpha (α)=0.12026.04 | 59.19 | 63.05 | — | — | — | |
| IsingClassifier=XGBoost2025.12 | 57.3 | 69.7 | — | 58.7 | 85.7 | |
| ChatGPTModality=TEXT, Instruction Tuning=false2024.02 | 56.6 | 57.6 | — | 61.2 | 54.4 | |
| FedFTGalpha (α)=0.052026.04 | 56.17 | 53.04 | — | — | — | |
| FedEnsemblealpha (α)=12026.04 | 55.98 | 68.63 | — | — | — | |
| FedGenalpha (α)=0.052026.04 | 55.78 | 53.02 | — | — | — | |
| FedAvgalpha (α)=0.52026.04 | 55.41 | 55.06 | — | — | — | |
| FEDACK-Aalpha (α)=0.052026.04 | 55.39 | 54.21 | — | — | — | |
| FedDFalpha (α)=0.12026.04 | 55.23 | — | — | — | — | |
| Mistral-7BModality=METADATA, Instruction Tuning=false2024.02 | 55.1 | 50.9 | — | 62.4 | 43 | |
| FedDistillalpha (α)=0.052026.04 | 54.3 | 52.5 | — | — | — | |
| FedEnsemblealpha (α)=0.12026.04 | 54.21 | 47.38 | — | — | — | |
| FedEnsemblealpha (α)=0.52026.04 | 54.15 | 45.73 | — | — | — | |
| FedEnsemblealpha (α)=0.052026.04 | 54.15 | 38.99 | — | — | — | |
| FedAvgalpha (α)=12026.04 | 54.04 | 53.37 | — | — | — | |
| LLAMA2-70BModality=TEXT, Instruction Tuning=false2024.02 | 53.9 | 58.5 | — | 57 | 60 | |
| FedDFalpha (α)=0.052026.04 | 53.35 | — | — | — | — | |
| BOTOMETERMetadata (M)=true, Texts (T)=false, Neighborhoods (N)=false2024.02 | 53.1 | 53.1 | — | 55.7 | 50.8 | |
| BotometerText Modality=false, Graph Modality=false, Modality Interaction=false2022.08 | 53.09 | 55.13 | — | — | — | |
| FedAvgalpha (α)=0.052026.04 | 52.46 | 50.41 | — | — | — | |
| FedProxalpha (α)=0.052026.04 | 52.3 | 51.02 | — | — | — | |
| FedProxalpha (α)=0.12026.04 | 51.86 | 50.08 | — | — | — | |
| FedAvgalpha (α)=0.12026.04 | 51.37 | 50.4 | — | — | — | |
| FedDistillalpha (α)=0.12026.04 | 50 | 50.08 | — | — | — | |
| Mistral-7BModality=TEXT, Instruction Tuning=false2024.02 | 49.1 | 39.8 | — | 55.3 | 31.1 |