Robustness Evaluation: Question Answering on ARC Easy
93.01Accuracy (After Attack)CacheTrap
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| CacheTrapModel=Qwen-2.5-3B, Target Class=02025.11 | 93.01 | — | 100 | |
| CacheTrapModel=Qwen-2.5-3B, Target Class=12025.11 | 93.01 | — | 100 | |
| CacheTrapModel=Qwen-2.5-3B, Target Class=22025.11 | 93.01 | — | 100 | |
| CacheTrapModel=Qwen-2.5-3B, Target Class=32025.11 | 93.01 | — | 100 | |
| CacheTrapModel=Qwen-2.5-3B, Target Class=42025.11 | 93.01 | — | 100 | |
| CacheTrapModel=LLaMA-3.1-8B, Target Class=02025.11 | 92.83 | — | 100 | |
| CacheTrapModel=LLaMA-3.1-8B, Target Class=12025.11 | 92.83 | — | 100 | |
| CacheTrapModel=LLaMA-3.1-8B, Target Class=22025.11 | 92.83 | — | 100 | |
| CacheTrapModel=LLaMA-3.1-8B, Target Class=32025.11 | 92.83 | — | 100 | |
| CacheTrapModel=LLaMA-3.1-8B, Target Class=42025.11 | 92.83 | — | 100 | |
| CacheTrapModel=Mistral-7B, Target Class=02025.11 | 89.69 | — | 100 | |
| CacheTrapModel=Mistral-7B, Target Class=12025.11 | 89.69 | — | 100 | |
| CacheTrapModel=Mistral-7B, Target Class=22025.11 | 89.69 | — | 100 | |
| CacheTrapModel=Mistral-7B, Target Class=32025.11 | 89.69 | — | 100 | |
| CacheTrapModel=Mistral-7B, Target Class=42025.11 | 89.69 | — | 100 | |
| CacheTrapModel=DeepSeek-7B, Target Class=02025.11 | 86.78 | — | 100 | |
| CacheTrapModel=DeepSeek-7B, Target Class=12025.11 | 86.78 | — | 100 | |
| CacheTrapModel=DeepSeek-7B, Target Class=22025.11 | 86.78 | — | 100 | |
| CacheTrapModel=DeepSeek-7B, Target Class=32025.11 | 86.78 | — | 100 | |
| CacheTrapModel=DeepSeek-7B, Target Class=42025.11 | 86.78 | — | 100 | |
| CacheTrapModel=LLaMA-2-7B, Target Class=02025.11 | 83.46 | — | 100 | |
| CacheTrapModel=LLaMA-2-7B, Target Class=12025.11 | 83.46 | — | 100 | |
| CacheTrapModel=LLaMA-2-7B, Target Class=22025.11 | 83.46 | — | 100 | |
| CacheTrapModel=LLaMA-2-7B, Target Class=32025.11 | 83.46 | — | 100 | |
| CacheTrapModel=LLaMA-2-7B, Target Class=42025.11 | 83.46 | — | 100 | |
| Quamba-2.8b-w4a16Bit-Flips=1, Attack=COBRA2025.12 | 60 | 68.7 | — | |
| GDF-BFABackbone=Qwen3-8B, # of bit flips=6, Quantization=INT82025.11 | 38.2 | 82.87 | — | |
| Mamba-370mBit-Flips=4, Attack=COBRA2025.12 | 28.67 | 53.125 | — | |
| GDF-BFABackbone=Mistral-7B-v0.3, # of bit flips=1, Quantization=INT82025.11 | 28.6 | 79.62 | — | |
| Llama2-13BModel=Llama2-13B, # of bit flips=3, Quantization precision=4-bit, Attack=GDF-BFA2025.11 | 27.8 | 77.56 | — | |
| GDF-BFABackbone=Llama2-7B, # of bit flips=1, Quantization=INT82025.11 | 27.18 | 75.79 | — | |
| Mamba-2.8bBit-Flips=1, Attack=COBRA2025.12 | 26.67 | 65.625 | — | |
| GDF-BFABackbone=Llama3-8B, # of bit flips=1, Quantization=INT82025.11 | 26.4 | 80.09 | — | |
| GDF-BFABackbone=Llama2-13B, # of bit flips=3, Quantization=INT82025.11 | 25.9 | 79.41 | — | |
| Llama3-8BModel=Llama3-8B, # of bit flips=1, Quantization precision=4-bit, Attack=GDF-BFA2025.11 | 25.16 | 78.5 | — | |
| Llama2-7BModel=Llama2-7B, # of bit flips=1, Quantization precision=4-bit, Attack=GDF-BFA2025.11 | 24.8 | 75.2 | — | |
| Quamba-1.4b-w8a8Bit-Flips=4, Attack=COBRA2025.12 | 24 | 63.15 | — | |
| Mamba-1.4bBit-Flips=1, Attack=COBRA2025.12 | 14 | 62.5 | — | |
| Mamba2-370mBit-Flips=4, Attack=COBRA2025.12 | 12.67 | 46.975 | — | |
| Mamba2-2.7bBit-Flips=9, Attack=COBRA2025.12 | 2 | 66.41 | — | |
| Quamba-1.4b-w4a16Bit-Flips=1, Attack=COBRA2025.12 | 0 | 61.75 | — | |
| Mamba2-1.3bBit-Flips=5, Attack=COBRA2025.12 | 0 | 62.5 | — | |
| Quamba-2.8b-w8a8Bit-Flips=1, Attack=COBRA2025.12 | 0 | 65.26 | — | |
| Hymba-1.5bBit-Flips=2, Attack=COBRA2025.12 | 0 | 76.94 | — |