Code-Specific Instruction Tuning Evaluation on Magicoder Evaluation Suite

54.27ARC-C AccuracyPre-trained

Evaluation results

Method
Pre-trained2026.05	54.27	80.09	72.62	86.2	73.3	60.98	67.14
Random2026.05	54.1	79.64	71.55	85.47	72.69	65.2	68.95
DualSFT2026.05	54.01	79.64	70.33	84.61	72.15	73.75	72.95
DualSFT-Data2026.05	53.66	80.03	70.75	85.04	72.37	74.21	73.29
DualSFT-Param2026.05	53.52	78.72	70.26	84.22	71.68	75.13	73.41
NanoAdam2026.05	52.87	79.03	70.22	84.22	71.59	73.28	72.43
S2FT2026.05	51.74	78.96	70.35	82.05	70.78	72.51	71.64
LoRA2026.05	51.51	76.62	69.47	81.94	69.89	70.29	70.09
Pre-trained2026.05	51.45	79.16	59.61	37	56.81	35.37	46.09
Random2026.05	51.25	78.89	56.91	34.65	55.43	49.24	52.33
LIFT2026.05	51.23	77.06	68.75	82.14	69.8	71.74	70.77
DualSFT2026.05	51.14	79.01	59.16	35.62	56.23	56.76	56.5
DualSFT-Param2026.05	51.13	78.54	58.14	36.34	56.04	57.52	56.78
DualSFT-Data2026.05	51.01	78.52	58.33	36.55	56.1	56.94	56.52
LESS + S2FT2026.05	51.01	78.66	55.68	31.14	54.12	53.12	53.62
LESS + S2FT2026.05	50.98	79.54	69.14	81.28	70.24	67.86	69.05
FisherSFT2026.05	50.94	76.16	68.49	81.36	69.24	71.61	70.42
LESS2026.05	50.92	78.92	54.14	31.58	53.89	57.09	55.49
SPICE2026.05	50.85	76.16	69.22	83.01	69.81	72.12	70.97
NanoAdam2026.05	50.78	78.2	58.33	33.12	55.11	56.38	55.74
SPICE + NanoAdam2026.05	50.78	78.54	57.29	33.89	55.13	52.65	53.89
SPICE + NanoAdam2026.05	50.55	79.35	70.09	80.74	70.18	69.24	69.71
SPICE2026.05	50.47	78.16	55.12	32.63	54.1	56.82	55.46
SMT2026.05	50.33	76.51	68.41	82.64	69.47	73.68	71.58
LESS2026.05	50.27	79.02	69.65	81.28	70.06	72.77	71.41
FisherSFT2026.05	50.25	74.51	53.71	32.17	52.66	53.47	53.07
LIFT2026.05	50.02	77.16	54.56	30.61	53.09	55.61	54.35
SMT2026.05	49.92	77.98	56.44	32.74	54.27	56.44	55.36
S2FT2026.05	49.74	78.29	55.75	31.09	53.72	55.29	54.5
Standard SFT2026.05	49.22	74.31	66.14	78.64	67.08	75.56	71.32
LoRA2026.05	49.15	75.57	56.71	31.63	53.27	54.66	53.96
Standard SFT2026.05	47.17	76.39	53.87	24.03	50.37	58.1	54.23
Pre-trained2026.05	43	76.61	56.52	27.45	50.9	28.66	39.78
Random2026.05	42.99	76.09	55.31	26.99	50.35	38.59	44.47
DualSFT2026.05	42.94	76.23	56.09	26.88	50.54	43.67	47.1
NanoAdam2026.05	42.66	74.68	52.71	24.39	48.61	43.54	46.08
DualSFT-Data2026.05	42.66	76.44	55.28	26.54	50.23	44.05	47.14
LoRA2026.05	42.61	74.43	51.87	23.29	48.05	41.46	44.76
LESS + S2FT2026.05	42.57	76.68	53.55	22.86	48.92	38.85	43.88
DualSFT-Param2026.05	42.54	76.19	54.01	26.27	49.75	44.52	47.14
LESS2026.05	42.18	75.66	53.97	23.46	48.82	42.66	45.74
SPICE + NanoAdam2026.05	42.16	76.61	54.87	25.93	49.89	39.52	44.71
S2FT2026.05	41.8	76.26	52.1	22.58	48.19	42.78	45.48
FisherSFT2026.05	41.55	74.17	52.83	22.92	47.87	41.37	44.62
LIFT2026.05	41.22	72.5	52.14	23.23	47.27	42.21	44.74
SMT2026.05	40.97	74.15	52.59	23.89	47.9	43.09	45.5
SPICE2026.05	40.54	76.09	53.13	24.52	48.57	42.32	45.45
Standard SFT2026.05	39.99	72.09	48.18	20.69	45.24	44.68	44.96