List your company
ResearchFormerly WizWand

Code Generation on CodeSearchNet (test)

0.8789BLEUOriginal Prompts
0.0200680.2430340.4660.688966Sep 19, 2024
Updated 8mo ago

Evaluation results

MethodLinks
Original PromptsCompression Rate=1.0, Compression Factor=1x2024.090.87890.92870.88690.91250.9861
TACO-RLCompression Rate=0.50, Compression Factor=2x, Optimization Objective=Bleu2024.090.35040.61260.38830.50770.9226
TACO-RLCompression Rate=0.50, Compression Factor=2x, Optimization Objective=Rouge12024.090.29970.59320.33990.47450.9176
TACO-RLCompression Rate=0.33, Compression Factor=3x, Optimization Objective=Bleu2024.090.28810.57320.33990.4690.9156
TACO-RLCompression Rate=0.25, Compression Factor=4x, Optimization Objective=Bleu2024.090.24230.54890.31260.44610.9114
LLMLinguaCompression Rate=0.50, Compression Factor=2x2024.090.23580.55230.28640.43410.9112
TACO-RLCompression Rate=0.33, Compression Factor=3x, Optimization Objective=Rouge12024.090.22770.54390.27340.41810.9083
TACO-RLCompression Rate=0.20, Compression Factor=5x, Optimization Objective=Bleu2024.090.21130.52670.29140.42840.9081
LLMLingua-2Compression Rate=0.50, Compression Factor=2x, Training Dataset=Py1502024.090.1990.52930.24350.39650.9061
TACO-RLCompression Rate=0.166, Compression Factor=6x, Optimization Objective=Bleu2024.090.18460.50990.27330.41340.9052
LLMLingua-2Compression Rate=0.50, Compression Factor=2x, Training Dataset=CodeSearchNet2024.090.1830.51570.2250.37940.9027
TACO-RLCompression Rate=0.25, Compression Factor=4x, Optimization Objective=Rouge12024.090.17590.50870.23030.38480.9022
TACO-RLCompression Rate=0.20, Compression Factor=5x, Optimization Objective=Rouge12024.090.14150.47940.2020.3610.8976
LLMLinguaCompression Rate=0.33, Compression Factor=3x2024.090.1340.46290.18210.34490.8933
TACO-RLCompression Rate=0.166, Compression Factor=6x, Optimization Objective=Rouge12024.090.12380.46070.18730.34930.8953
LLMLingua-2Compression Rate=0.33, Compression Factor=3x, Training Dataset=Py1502024.090.12110.44940.15810.3210.891
LLMLingua-2Compression Rate=0.33, Compression Factor=3x, Training Dataset=CodeSearchNet2024.090.11980.44950.15830.32070.8906
LLMLingua-2Compression Rate=0.25, Compression Factor=4x, Training Dataset=Py1502024.090.09350.41620.12810.29210.8845
LLMLinguaCompression Rate=0.25, Compression Factor=4x2024.090.09060.41510.13690.30150.8834
LLMLingua-2Compression Rate=0.25, Compression Factor=4x, Training Dataset=CodeSearchNet2024.090.08930.41070.1230.28870.8831
LLMLingua-2Compression Rate=0.20, Compression Factor=5x, Training Dataset=Py1502024.090.07760.39090.110.27460.88
LLMLingua-2Compression Rate=0.20, Compression Factor=5x, Training Dataset=CodeSearchNet2024.090.07310.38880.10560.27210.8787
LLMLingua-2Compression Rate=0.166, Compression Factor=6x, Training Dataset=Py1502024.090.06790.37630.09950.26490.8769
LLMLinguaCompression Rate=0.20, Compression Factor=5x2024.090.06570.38630.11130.27810.8774
LLMLingua-2Compression Rate=0.166, Compression Factor=6x, Training Dataset=CodeSearchNet2024.090.06540.37280.09580.2620.8764
LLMLinguaCompression Rate=0.166, Compression Factor=6x2024.090.05310.36620.09760.26380.8736