Sentiment Classification on SST2
96AccuracyPrivateLoRA
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| PrivateLoRALLM Type=Open, Model=Llama3-8B(Instruct), Privacy Budget (epsilon)=8, Training Cost ($)=27.6, Query Cost ($)=0.78, Total Cost ($)=28.382024.11 | 96 | 90.2 | — | |
| DP-ICLLLM Type=Closed, Model=GPT4 Turbo, Privacy Budget (epsilon)=8, Training Cost ($)=0, Query Cost ($)=138, Total Cost ($)=1382024.11 | 95.9 | 68.2 | — | |
| PromptPATELLM Type=Closed, Model=Claude 2.1, Privacy Budget (epsilon)=< 0.3, Training Cost ($)=48.24, Query Cost ($)=5.36, Total Cost ($)=53.62024.11 | 95.7 | 84.5 | — | |
| PrivateLoRALLM Type=Open, Model=Vicuna 7B, Privacy Budget (epsilon)=8, Training Cost ($)=13.8, Query Cost ($)=0.78, Total Cost ($)=14.582024.11 | 94.8 | 90.3 | — | |
| ICLLLM Type=Closed, Model=GPT3 Davinci, Privacy Budget (epsilon)=infinity, Training Cost ($)=0, Query Cost ($)=6, Total Cost ($)=62024.11 | 94.7 | 82.9 | — | |
| PromptPATELLM Type=Closed, Model=GPT3 Babbage, Privacy Budget (epsilon)=8, Training Cost ($)=8.66, Query Cost ($)=1.72, Total Cost ($)=10.382024.11 | 93.8 | 75 | — | |
| PrivateLoRALLM Type=Open, Model=ROBERTA Large, Privacy Budget (epsilon)=8, Training Cost ($)=3.45, Query Cost ($)=0.4, Total Cost ($)=3.852024.11 | 93.6 | 89.3 | — | |
| DP2O2024.06 | 93.6 | — | 0.7 | |
| DP-FineTuneLLM Type=Open, Model=ROBERTA Large, Privacy Budget (epsilon)=8, Training Cost ($)=5.75, Query Cost ($)=0.4, Total Cost ($)=6.152024.11 | 93.5 | 89.4 | — | |
| DP-ICLLLM Type=Closed, Model=GPT3 Babbage, Privacy Budget (epsilon)=8, Training Cost ($)=0, Query Cost ($)=17.2, Total Cost ($)=17.22024.11 | 92.8 | 62.6 | — | |
| 0-shotLLM Type=Closed, Model=GPT3 Davinci, Privacy Budget (epsilon)=0, Training Cost ($)=0, Query Cost ($)=6, Total Cost ($)=62024.11 | 92.4 | 76.3 | — | |
| PromptDPSGDLLM Type=Open, Model=ROBERTA Large, Privacy Budget (epsilon)=8, Training Cost ($)=7.59, Query Cost ($)=0.4, Total Cost ($)=7.992024.11 | 92.3 | 68.6 | — | |
| DP-OPT (original)LLM Type=Closed, Model=GPT3 Davinci, Privacy Budget (epsilon)=8, Training Cost ($)=2.1, Query Cost ($)=6, Total Cost ($)=8.12024.11 | 92.2 | 81.4 | — | |
| PrivateLoRALLM Type=Open, Model=Pythia 6.9B, Privacy Budget (epsilon)=8, Training Cost ($)=13.8, Query Cost ($)=0.78, Total Cost ($)=14.582024.11 | 92.2 | 89.4 | — | |
| MPD2024.06 | 92.1 | — | 0.1 | |
| TEMPERA2024.06 | 91.9 | — | 2 | |
| DP-OPT (local)LLM Type=Open, Model=Vicuna 7B, Privacy Budget (epsilon)=8, Training Cost ($)=2.1, Query Cost ($)=0.78, Total Cost ($)=2.882024.11 | 89.5 | 75.3 | — | |
| PromptPATELLM Type=Closed, Model=GPT3 Babbage, Privacy Budget (epsilon)=< 0.3, Training Cost ($)=9.72, Query Cost ($)=1.72, Total Cost ($)=11.442024.11 | 88.8 | 69.6 | — | |
| Uncompressed LSTMModel=LSTM, R(%)=0, Size(MB)=53.252018.11 | 86.03 | — | — | |
| Low Rank Matrix FactorizationModel=LSTM, R(%)=10, Size(MB)=48.662018.11 | 85.72 | — | — | |
| Low Rank Matrix FactorizationModel=LSTM, R(%)=30, Size(MB)=38.522018.11 | 85.68 | — | — | |
| Low Rank Matrix FactorizationModel=LSTM, R(%)=50, Size(MB)=28.452018.11 | 85.67 | — | — | |
| Low Rank Matrix FactorizationModel=LSTM, R(%)=70, Size(MB)=18.382018.11 | 85.45 | — | — | |
| Offline Compression (Baseline 2)Model=LSTM, R(%)=10, Size(MB)=48.662018.11 | 85.45 | — | — | |
| Low Rank Matrix FactorizationModel=LSTM, R(%)=90, Size(MB)=6.942018.11 | 85.11 | — | — | |
| Quantized LSTM (Baseline 1)Model=LSTM, R(%)=50, Size(MB)=30.16, Bit-width=16 bit2018.11 | 85.08 | — | — | |
| Quantized LSTM (Baseline 1)Model=LSTM, R(%)=75, Size(MB)=18.21, Bit-width=8 bit2018.11 | 85.01 | — | — | |
| Offline Compression (Baseline 2)Model=LSTM, R(%)=30, Size(MB)=38.522018.11 | 84.95 | — | — | |
| Uncompressed DANModel=DAN (DAN-RAND variant), R(%)=0, Size(MB)=52.842018.11 | 84.61 | — | — | |
| Offline Compression (Baseline 2)Model=LSTM, R(%)=50, Size(MB)=28.452018.11 | 84.24 | — | — | |
| Low Rank Matrix FactorizationModel=DAN (DAN-RAND variant), R(%)=10, Size(MB)=47.292018.11 | 84.24 | — | — | |
| Low Rank Matrix FactorizationModel=DAN (DAN-RAND variant), R(%)=30, Size(MB)=37.232018.11 | 83.83 | — | — | |
| Low Rank Matrix FactorizationModel=DAN (DAN-RAND variant), R(%)=50, Size(MB)=27.162018.11 | 83.72 | — | — | |
| Low Rank Matrix FactorizationModel=DAN (DAN-RAND variant), R(%)=70, Size(MB)=17.182018.11 | 83.67 | — | — | |
| Offline Compression (Baseline 2)Model=DAN (DAN-RAND variant), R(%)=10, Size(MB)=47.292018.11 | 83.47 | — | — | |
| Offline Compression (Baseline 2)Model=DAN (DAN-RAND variant), R(%)=30, Size(MB)=37.232018.11 | 83.31 | — | — | |
| Quantized DAN (Baseline 1)Model=DAN (DAN-RAND variant), R(%)=50, Size(MB)=28.36, Bit-width=16 bit2018.11 | 83.18 | — | — | |
| Low Rank Matrix FactorizationModel=DAN (DAN-RAND variant), R(%)=90, Size(MB)=6.212018.11 | 83.11 | — | — | |
| Offline Compression (Baseline 2)Model=LSTM, R(%)=70, Size(MB)=18.382018.11 | 83.09 | — | — | |
| Quantized DAN (Baseline 1)Model=DAN (DAN-RAND variant), R(%)=75, Size(MB)=18.13, Bit-width=8 bit2018.11 | 82.94 | — | — | |
| Offline Compression (Baseline 2)Model=DAN (DAN-RAND variant), R(%)=50, Size(MB)=27.162018.11 | 82.87 | — | — | |
| Offline Compression (Baseline 2)Model=DAN (DAN-RAND variant), R(%)=70, Size(MB)=17.182018.11 | 82.86 | — | — | |
| Offline Compression (Baseline 2)Model=DAN (DAN-RAND variant), R(%)=90, Size(MB)=6.212018.11 | 82.59 | — | — | |
| Offline Compression (Baseline 2)Model=LSTM, R(%)=90, Size(MB)=6.942018.11 | 82.54 | — | — | |
| PrivateLoRALLM Type=Open, Model=Pythia 160M, Privacy Budget (epsilon)=8, Training Cost ($)=1.6, Query Cost ($)=0.5, Total Cost ($)=2.12024.11 | 80.4 | 78.6 | — | |
| DP-FewShotGenLLM Type=Closed, Model=GPT3 Babbage, Privacy Budget (epsilon)=8, Training Cost ($)=0.86, Query Cost ($)=1.1, Total Cost ($)=1.962024.11 | 72.8 | 64.2 | — |