Question Answering on MuSiQuest-E 2022 (Full)
46F1 ScoreKVComm
Evaluation Results
| Method | Links | |
|---|---|---|
| KVCommSender Model (Ms)=Orion-ehristoforu/falcon3-ultraset, Receiver Model (Mr)=huihui-ai/Falcon3-7B-Instruct-abliterated, compression_rate=0.52025.10 | 46 | |
| KVCommSender Model (Ms)=Orion-ehristoforu/falcon3-ultraset, Receiver Model (Mr)=huihui-ai/Falcon3-7B-Instruct-abliterated, compression_rate=0.32025.10 | 40 | |
| KVCommSender Model (Ms)=Orion-ehristoforu/falcon3-ultraset, Receiver Model (Mr)=huihui-ai/Falcon3-7B-Instruct-abliterated, compression_rate=0.72025.10 | 36 | |
| SkylineSender Model (Ms)=Orion-ehristoforu/falcon3-ultraset, Receiver Model (Mr)=huihui-ai/Falcon3-7B-Instruct-abliterated2025.10 | 33 | |
| KVCommSender Model (Ms)=huihui-ai/Llama-3.2-3B-Instruct-abliterated, Receiver Model (Mr)=suayptalha/DeepSeek-R1-Distill-Llama-3B, compression_rate=0.72025.10 | 30 | |
| NLDSender Model (Ms)=Orion-ehristoforu/falcon3-ultraset, Receiver Model (Mr)=huihui-ai/Falcon3-7B-Instruct-abliterated2025.10 | 28 | |
| SkylineSender Model (Ms)=huihui-ai/Llama-3.2-3B-Instruct-abliterated, Receiver Model (Mr)=suayptalha/DeepSeek-R1-Distill-Llama-3B2025.10 | 25 | |
| KVCommSender Model (Ms)=huihui-ai/Llama-3.2-3B-Instruct-abliterated, Receiver Model (Mr)=suayptalha/DeepSeek-R1-Distill-Llama-3B, compression_rate=0.52025.10 | 25 | |
| KVCommSender Model (Ms)=Orion-zhen/Qwen2.5-7B-Instruct-Uncensored, Receiver Model (Mr)=bespokelabs/Bespoke-Stratos-7B, compression_rate=0.72025.10 | 21 | |
| CIPHERSender Model (Ms)=huihui-ai/Llama-3.2-3B-Instruct-abliterated, Receiver Model (Mr)=suayptalha/DeepSeek-R1-Distill-Llama-3B2025.10 | 20 | |
| NLDSender Model (Ms)=huihui-ai/Llama-3.2-3B-Instruct-abliterated, Receiver Model (Mr)=suayptalha/DeepSeek-R1-Distill-Llama-3B2025.10 | 18 | |
| KVCommSender Model (Ms)=huihui-ai/Llama-3.2-3B-Instruct-abliterated, Receiver Model (Mr)=suayptalha/DeepSeek-R1-Distill-Llama-3B, compression_rate=0.32025.10 | 11 | |
| SkylineSender Model (Ms)=Orion-zhen/Qwen2.5-7B-Instruct-Uncensored, Receiver Model (Mr)=bespokelabs/Bespoke-Stratos-7B2025.10 | 10 | |
| CIPHERSender Model (Ms)=Orion-ehristoforu/falcon3-ultraset, Receiver Model (Mr)=huihui-ai/Falcon3-7B-Instruct-abliterated2025.10 | 9 | |
| KVCommSender Model (Ms)=Orion-zhen/Qwen2.5-7B-Instruct-Uncensored, Receiver Model (Mr)=bespokelabs/Bespoke-Stratos-7B, compression_rate=0.52025.10 | 8 | |
| NLDSender Model (Ms)=Orion-zhen/Qwen2.5-7B-Instruct-Uncensored, Receiver Model (Mr)=bespokelabs/Bespoke-Stratos-7B2025.10 | 7 | |
| AC (mean)Sender Model (Ms)=Orion-ehristoforu/falcon3-ultraset, Receiver Model (Mr)=huihui-ai/Falcon3-7B-Instruct-abliterated2025.10 | 7 | |
| BaselineSender Model (Ms)=huihui-ai/Llama-3.2-3B-Instruct-abliterated, Receiver Model (Mr)=suayptalha/DeepSeek-R1-Distill-Llama-3B2025.10 | 6 | |
| AC (mean)Sender Model (Ms)=huihui-ai/Llama-3.2-3B-Instruct-abliterated, Receiver Model (Mr)=suayptalha/DeepSeek-R1-Distill-Llama-3B2025.10 | 6 | |
| AC (sum)Sender Model (Ms)=huihui-ai/Llama-3.2-3B-Instruct-abliterated, Receiver Model (Mr)=suayptalha/DeepSeek-R1-Distill-Llama-3B2025.10 | 6 | |
| BaselineSender Model (Ms)=Orion-zhen/Qwen2.5-7B-Instruct-Uncensored, Receiver Model (Mr)=bespokelabs/Bespoke-Stratos-7B2025.10 | 6 | |
| BaselineSender Model (Ms)=Orion-ehristoforu/falcon3-ultraset, Receiver Model (Mr)=huihui-ai/Falcon3-7B-Instruct-abliterated2025.10 | 6 | |
| AC (sum)Sender Model (Ms)=Orion-ehristoforu/falcon3-ultraset, Receiver Model (Mr)=huihui-ai/Falcon3-7B-Instruct-abliterated2025.10 | 6 | |
| CIPHERSender Model (Ms)=Orion-zhen/Qwen2.5-7B-Instruct-Uncensored, Receiver Model (Mr)=bespokelabs/Bespoke-Stratos-7B2025.10 | 2 | |
| AC (replace)Sender Model (Ms)=huihui-ai/Llama-3.2-3B-Instruct-abliterated, Receiver Model (Mr)=suayptalha/DeepSeek-R1-Distill-Llama-3B2025.10 | 1 | |
| KVCommSender Model (Ms)=Orion-zhen/Qwen2.5-7B-Instruct-Uncensored, Receiver Model (Mr)=bespokelabs/Bespoke-Stratos-7B, compression_rate=0.32025.10 | 1 | |
| AC (replace)Sender Model (Ms)=Orion-ehristoforu/falcon3-ultraset, Receiver Model (Mr)=huihui-ai/Falcon3-7B-Instruct-abliterated2025.10 | 1 | |
| AC (mean)Sender Model (Ms)=Orion-zhen/Qwen2.5-7B-Instruct-Uncensored, Receiver Model (Mr)=bespokelabs/Bespoke-Stratos-7B2025.10 | 0 | |
| AC (replace)Sender Model (Ms)=Orion-zhen/Qwen2.5-7B-Instruct-Uncensored, Receiver Model (Mr)=bespokelabs/Bespoke-Stratos-7B2025.10 | 0 | |
| AC (sum)Sender Model (Ms)=Orion-zhen/Qwen2.5-7B-Instruct-Uncensored, Receiver Model (Mr)=bespokelabs/Bespoke-Stratos-7B2025.10 | 0 |