Classification on BANKCHURN (test)
82.7Average AccuracyACT
Evaluation results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| ACTModel=Gemma, d=8, k=102025.09 | 82.7 | — | — | — | — | — | — | — | — | 40.6 | |
| ACTModel=Nano, d=4, k=202025.09 | 80.7 | — | — | — | — | — | — | — | — | 9.1 | |
| ACTModel=Nano, d=8, k=102025.09 | 80.7 | — | — | — | — | — | — | — | — | 12.4 | |
| TF-IDF + CARTd=32025.09 | 80.7 | — | — | — | — | — | — | — | — | 4.5 | |
| TextGradModel=Nano2025.09 | 80.5 | — | — | — | — | — | — | — | — | 0 | |
| DSPyModel=Nano2025.09 | 80.3 | — | — | — | — | — | — | — | — | 0 | |
| TextGradModel=Gemma2025.09 | 78.9 | — | — | — | — | — | — | — | — | 2.4 | |
| ACTModel=Gemma, d=4, k=202025.09 | 75 | — | — | — | — | — | — | — | — | 22.9 | |
| RoBERTa2025.09 | 72.3 | — | — | — | — | — | — | — | — | 72.3 | |
| BERT2025.09 | 71.7 | — | — | — | — | — | — | — | — | 71.6 | |
| ACTTree Depth (d)=4, Optimization Steps (k)=20, LLM Backbone=GPT-4.1 Mini2025.09 | 68.7 | — | — | — | — | — | — | — | — | 68.6 | |
| CoTModel=Nano2025.09 | 67.8 | — | — | — | — | — | — | — | — | 19.1 | |
| ACTtree depth (d)=4, optimization steps (k)=202025.09 | 65.2 | 65.2 | 70.7 | 65 | 69.1 | 68.7 | 68.6 | 62 | 60 | 67.1 | |
| ACTTree Depth (d)=4, Optimization Steps (k)=20, LLM Backbone=Gemma3 4b2025.09 | 65.2 | — | — | — | — | — | — | — | — | 70.7 | |
| ACTTree Depth (d)=4, Optimization Steps (k)=20, LLM Backbone=Avg2025.09 | 65.2 | — | — | — | — | — | — | — | — | 67.1 | |
| ACTTree Depth (d)=4, Optimization Steps (k)=20, LLM Backbone=GPT-4.1 Nano2025.09 | 65 | — | — | — | — | — | — | — | — | 69.1 | |
| ACTtree depth (d)=3, optimization steps (k)=202025.09 | 64.2 | 58 | 47.8 | 67 | 67.5 | 69.2 | 65.3 | 62.7 | 63.7 | 61.1 | |
| ACTtree depth (d)=4, optimization steps (k)=102025.09 | 63.3 | 61.7 | 65.3 | 61.7 | 60.9 | 69 | 65 | 60.9 | 60.2 | 62.9 | |
| TF-IDF + CARTtree depth (d)=52025.09 | 63.2 | — | — | — | — | — | — | — | — | 66.8 | |
| TF-IDF + CARTTree Depth (d)=5, LLM Backbone=Avg2025.09 | 63.2 | — | — | — | — | — | — | — | — | 66.8 | |
| Rule Fit2025.09 | 63 | — | — | — | — | — | — | — | — | 59.8 | |
| Rule FitLLM Backbone=Avg2025.09 | 63 | — | — | — | — | — | — | — | — | 59.8 | |
| ACTTree Depth (d)=4, Optimization Steps (k)=20, LLM Backbone=Qwen3 4b2025.09 | 62 | — | — | — | — | — | — | — | — | 60 | |
| ACTTree Depth (d)=3, Optimization Steps (k)=10, LLM Backbone=GPT-4.1 Mini2025.09 | 60.5 | — | — | — | — | — | — | — | — | 64.6 | |
| Con. tag.LLM Backbone=GPT-4.1 Mini2025.09 | 59.7 | — | — | — | — | — | — | — | — | 60.8 | |
| ACTTree Depth (d)=3, Optimization Steps (k)=10, LLM Backbone=GPT-4.1 Nano2025.09 | 58.8 | — | — | — | — | — | — | — | — | 67.4 | |
| ACTtree depth (d)=3, optimization steps (k)=102025.09 | 58.3 | 56.3 | 42.3 | 58.8 | 67.4 | 60.5 | 64.6 | 57.5 | 62.7 | 59.3 | |
| ACTTree Depth (d)=3, Optimization Steps (k)=10, LLM Backbone=Avg2025.09 | 58.3 | — | — | — | — | — | — | — | — | 59.3 | |
| ACTTree Depth (d)=3, Optimization Steps (k)=10, LLM Backbone=Qwen3 4b2025.09 | 57.5 | — | — | — | — | — | — | — | — | 62.7 | |
| DSPyEvaluation Protocol=BFSR, 8 demos, LLM Backbone=GPT-4.1 Mini2025.09 | 57.2 | — | — | — | — | — | — | — | — | 58.5 | |
| Con. tag.LLM Backbone=Gemma3 4b2025.09 | 57.2 | — | — | — | — | — | — | — | — | 60.2 | |
| TF-IDF + XGBoosttree depth (d)=2, number of trees (nt)=2002025.09 | 56.7 | — | — | — | — | — | — | — | — | 48.8 | |
| ACTTree Depth (d)=3, Optimization Steps (k)=10, LLM Backbone=Gemma3 4b2025.09 | 56.3 | — | — | — | — | — | — | — | — | 42.3 | |
| DSPyModel=Gemma2025.09 | 56 | — | — | — | — | — | — | — | — | 33.2 | |
| Con. tag.2025.09 | 55.8 | 57.2 | 60.2 | 52 | 59.8 | 59.7 | 60.8 | 54.3 | 61.6 | 60.6 | |
| Con. tag.LLM Backbone=Avg2025.09 | 55.8 | — | — | — | — | — | — | — | — | 60.6 | |
| TEXTGRADLLM Backbone=GPT-4.1 Mini2025.09 | 55.3 | — | — | — | — | — | — | — | — | 61 | |
| Con. tag.LLM Backbone=Qwen3 4b2025.09 | 54.3 | — | — | — | — | — | — | — | — | 61.6 | |
| DSPysearch strategy=BFSR, number of demos=82025.09 | 54.1 | 52.8 | 55.6 | 52.3 | 42.8 | 57.2 | 58.5 | 54 | 52.2 | 52.3 | |
| DSPyEvaluation Protocol=BFSR, 8 demos, LLM Backbone=Avg2025.09 | 54.1 | — | — | — | — | — | — | — | — | 52.3 | |
| DSPyEvaluation Protocol=BFSR, 8 demos, LLM Backbone=Qwen3 4b2025.09 | 54 | — | — | — | — | — | — | — | — | 52.2 | |
| TEXTGRAD2025.09 | 53.4 | 52 | 60 | 53.3 | 60.3 | 55.3 | 61 | 52.8 | 61.4 | 60.7 | |
| TEXTGRADLLM Backbone=Avg2025.09 | 53.4 | — | — | — | — | — | — | — | — | 60.7 | |
| TEXTGRADLLM Backbone=GPT-4.1 Nano2025.09 | 53.3 | — | — | — | — | — | — | — | — | 60.3 | |
| CoTEvaluation Protocol=0-shot, LLM Backbone=GPT-4.1 Mini2025.09 | 52.8 | — | — | — | — | — | — | — | — | 60.5 | |
| DSPyEvaluation Protocol=BFSR, 8 demos, LLM Backbone=Gemma3 4b2025.09 | 52.8 | — | — | — | — | — | — | — | — | 55.6 | |
| TEXTGRADLLM Backbone=Qwen3 4b2025.09 | 52.8 | — | — | — | — | — | — | — | — | 61.4 | |
| DSPyEvaluation Protocol=BFSR, 8 demos, LLM Backbone=GPT-4.1 Nano2025.09 | 52.3 | — | — | — | — | — | — | — | — | 42.8 | |
| TEXTGRADLLM Backbone=Gemma3 4b2025.09 | 52 | — | — | — | — | — | — | — | — | 60 | |
| Con. tag.LLM Backbone=GPT-4.1 Nano2025.09 | 52 | — | — | — | — | — | — | — | — | 59.8 | |
| CoTEvaluation Protocol=0-shot, LLM Backbone=Qwen3 4b2025.09 | 50.2 | — | — | — | — | — | — | — | — | 37.3 | |
| CoTshot configuration=0-shot2025.09 | 49.8 | 47.8 | 48.1 | 48.5 | 25.9 | 52.8 | 60.5 | 50.2 | 37.3 | 43 | |
| CoTEvaluation Protocol=0-shot, LLM Backbone=Avg2025.09 | 49.8 | — | — | — | — | — | — | — | — | 43 | |
| CoTEvaluation Protocol=0-shot, LLM Backbone=GPT-4.1 Nano2025.09 | 48.5 | — | — | — | — | — | — | — | — | 25.9 | |
| CoTEvaluation Protocol=0-shot, LLM Backbone=Gemma3 4b2025.09 | 47.8 | — | — | — | — | — | — | — | — | 48.1 | |
| CoTModel=Gemma2025.09 | 47.3 | — | — | — | — | — | — | — | — | 27.3 |