List your company
ResearchFormerly WizWand

Question Answering on ARC-Easy (test)

97.2AccuracyMLP Probe
18.742439.111259.4879.8488May 24, 2025Jul 22, 2025Sep 20, 2025Nov 19, 2025Jan 18, 2026Mar 19, 2026May 18, 2026
Updated 5mo ago

Evaluation results

MethodLinks
MLP ProbeEvaluation Protocol=Probe2026.0297.2—
Teacher 5-shotEvaluation Protocol=5-shot2026.0296.6—
PROBE-KD (MLP)Backbone=DeBERTa-v3-base, Parameters=86M, Probe Type=MLP2026.0275.1—
Patient-KDBackbone=DeBERTa-v3-base, Parameters=86M, Distillation Strategy=Patient-KD2026.0274.6—
Logit-KDBackbone=DeBERTa-v3-base, Parameters=86M, Distillation Strategy=Logit-KD2026.0274.4—
PROBE-KD (Logistic)Backbone=DeBERTa-v3-base, Parameters=86M, Probe Type=Logistic2026.0274.3—
PROBE-KD (CCS)Backbone=DeBERTa-v3-base, Parameters=86M, Probe Type=CCS2026.0274.1—
Label SmoothingBackbone=DeBERTa-v3-base, Parameters=86M2026.0273.9—
SupervisedBackbone=DeBERTa-v3-base, Parameters=86M2026.0273.5—
Feature-KDBackbone=DeBERTa-v3-base, Parameters=86M, Distillation Strategy=Feature-KD2026.0269.7—
AMOOptimizer=AMO, Model=Llama3.1-1.4B, Few-shot=02026.0566.46—
Muon-PEOptimizer=Muon-PE, Model=Llama3.1-1.4B, Few-shot=02026.0566.41—
AdamWOptimizer=AdamW, Model=Llama3.1-1.4B, Few-shot=02026.0565.84—
ROOTOptimizer=ROOT, Model=Llama3.1-1.4B, Few-shot=02026.0565.74—
Muon-CANSOptimizer=Muon-CANS, Model=Llama3.1-1.4B, Few-shot=02026.0565.57—
Muon-KJOptimizer=Muon-KJ, Model=Llama3.1-1.4B, Few-shot=02026.0565.32—
Muon-YouOptimizer=Muon-You, Model=Llama3.1-1.4B, Few-shot=02026.0564.81—
Muon-TurboOptimizer=Muon-Turbo, Model=Llama3.1-1.4B, Few-shot=02026.0564.77—
AdaMuonOptimizer=AdaMuon, Model=Llama3.1-1.4B, Few-shot=02026.0563.17—
Muon-YouOptimizer=Muon-You, Model=Llama3.1-760M, Few-shot=02026.0560.27—
AdamWOptimizer=AdamW, Model=Llama3.1-760M, Few-shot=02026.0559.89—
Muon-PEOptimizer=Muon-PE, Model=Llama3.1-760M, Few-shot=02026.0559.89—
ROOTOptimizer=ROOT, Model=Llama3.1-760M, Few-shot=02026.0559.85—
Muon-TurboOptimizer=Muon-Turbo, Model=Llama3.1-760M, Few-shot=02026.0559.81—
Muon-KJOptimizer=Muon-KJ, Model=Llama3.1-760M, Few-shot=02026.0559.68—
Muon-CANSOptimizer=Muon-CANS, Model=Llama3.1-760M, Few-shot=02026.0559.51—
AMOOptimizer=AMO, Model=Llama3.1-760M, Few-shot=02026.0559.43—
AdaMuonOptimizer=AdaMuon, Model=Llama3.1-760M, Few-shot=02026.0558.29—
GQA (Original)Rank=64, KV Save=93.75, Model=Qwen3-4B-Instruct-25072026.0355.89—
CARE-U (OURS)Rank=512, KV Save=50.00, Model=Llama-3.1-8B-Instruct2026.0352.73—
SVD-LLM V2Rank=512, KV Save=50.00, Model=Llama-3.1-8B-Instruct2026.0352.39—
GQA (Original)Rank=64, KV Save=93.75, Model=Llama-3.1-8B-Instruct2026.0350.34—
MuSeModel Scale=1B, Training Data=scientific PDFs2025.0948.743.6
CUDNNModel Scale=1B, Training Data=scientific PDFs2025.0947.943.1
ASVDRank=512, KV Save=50.00, Model=Llama-3.1-8B-Instruct2026.0346.33—
CARE-E (OURS)Rank=512, KV Save=50.00, Model=Llama-3.1-8B-Instruct2026.0342.41—
PGM 8 / 8distillation=Before2025.0540.4—
PGM 6 / 6 (1024)distillation=Before2025.0539.98—
Baseline GPT-2shot=0-shot, protocol=standalone evaluation2026.0438.9335.61
CARE-E (OURS)Rank=256, KV Save=75.00, Model=Llama-3.1-8B-Instruct2026.0338.48—
MDLMdistillation=Before2025.0537.88—
PGM 8 / 8distillation=After Distillation (SDTT)2025.0537.42—
GMT v7shot=0-shot, protocol=standalone evaluation2026.0437.0433.84
PGM 6 / 6 (1024)distillation=After Distillation (SDTT)2025.0536.7—
SVD-LLM V2Rank=256, KV Save=75.00, Model=Llama-3.1-8B-Instruct2026.0336.26—
CARE-U (OURS)Rank=256, KV Save=75.00, Model=Llama-3.1-8B-Instruct2026.0336.09—
MDLMdistillation=After Distillation (SDTT)2025.0533.8—
CARE-E (OURS)Rank=128, KV Save=87.50, Model=Qwen3-4B-Instruct-25072026.0330.29—
Palu(SVD)Rank=512, KV Save=50.00, Model=Llama-3.1-8B-Instruct2026.0328.16—
SVD-LLM V2Rank=128, KV Save=87.50, Model=Llama-3.1-8B-Instruct2026.0327.56—
MHA2MLARank=256, KV Save=75.00, Model=Llama-3.1-8B-Instruct2026.0327.47—
CARE-U (OURS)Rank=128, KV Save=87.50, Model=Qwen3-4B-Instruct-25072026.0327.47—
ASVDRank=64, KV Save=93.75, Model=Qwen3-4B-Instruct-25072026.0327.39—
CARE-E (OURS)Rank=128, KV Save=87.50, Model=Llama-3.1-8B-Instruct2026.0327.3—
CARE-U (OURS)Rank=128, KV Save=87.50, Model=Llama-3.1-8B-Instruct2026.0326.71—
Palu(SVD)Rank=128, KV Save=87.50, Model=Qwen3-4B-Instruct-25072026.0326.02—
MHA2MLARank=64, KV Save=93.75, Model=Llama-3.1-8B-Instruct2026.0325.94—
MHA2MLARank=512, KV Save=50.00, Model=Llama-3.1-8B-Instruct2026.0325.94—
Palu(SVD)Rank=64, KV Save=93.75, Model=Llama-3.1-8B-Instruct2026.0325.77—
MHA2MLARank=64, KV Save=93.75, Model=Qwen3-4B-Instruct-25072026.0325.68—
Palu(SVD)Rank=64, KV Save=93.75, Model=Qwen3-4B-Instruct-25072026.0325.34—
MHA2MLARank=128, KV Save=87.50, Model=Llama-3.1-8B-Instruct2026.0325—
ASVDRank=128, KV Save=87.50, Model=Llama-3.1-8B-Instruct2026.0325—
CARE-U (OURS)Rank=64, KV Save=93.75, Model=Llama-3.1-8B-Instruct2026.0323.89—
Palu(SVD)Rank=128, KV Save=87.50, Model=Llama-3.1-8B-Instruct2026.0323.89—
ASVDRank=64, KV Save=93.75, Model=Llama-3.1-8B-Instruct2026.0323.81—
MHA2MLARank=128, KV Save=87.50, Model=Qwen3-4B-Instruct-25072026.0323.81—
CARE-E (OURS)Rank=64, KV Save=93.75, Model=Llama-3.1-8B-Instruct2026.0323.63—
SVD-LLM V2Rank=64, KV Save=93.75, Model=Llama-3.1-8B-Instruct2026.0323.63—
CARE-U (OURS)Rank=64, KV Save=93.75, Model=Qwen3-4B-Instruct-25072026.0323.46—
CARE-E (OURS)Rank=64, KV Save=93.75, Model=Qwen3-4B-Instruct-25072026.0323.29—
SVD-LLM V2Rank=64, KV Save=93.75, Model=Qwen3-4B-Instruct-25072026.0323.12—
Palu(SVD)Rank=256, KV Save=75.00, Model=Llama-3.1-8B-Instruct2026.0322.61—
ASVDRank=256, KV Save=75.00, Model=Llama-3.1-8B-Instruct2026.0321.76—