List your company

NLU and Reasoning on BIG-bench (unseen)

86.96Known Unknowns ScoreFLIPPED
46.264856.829967.39577.9601Oct 6, 2022Oct 26, 2022Nov 16, 2022Dec 7, 2022Dec 27, 2022Jan 17, 2023Feb 7, 2023

Evaluation results

MethodLinks
FLIPPEDParameters=11B, Setting=Zero-shot, Shots=02022.1086.9642.553.2352.5748.496064.9326.8765.5750.3954.3464.5655.0855.1746.88
FLIPPEDParameters=3B, Setting=Zero-shot, Shots=02022.1071.7441.753.1947.4347.934561.6419.0157.0750.5646.5859.8253.3348.5725
PALMParameters=540B, Setting=Few-shot, Shots=12022.1067.3942.26994.8657.261.6780.0237.370.452.277.6334.4247.560.859.38
T0Parameters=11B, Training=Instruction Tuning2023.0265.2233.6754.6742.8657.3351.6771.7218.3357.3348.3352.9754.675551.06—
DIRECTParameters=3B, Setting=Zero-shot, Shots=02022.1063.0435.953.2850.2947.1533.3358.1422.0163.8349.8550.2338.0644.3343.753.13
GPT-3Parameters=175B2023.0260.8731.252.322.5751.431.6726.7815.912.350.547.9523.4251.537.57—
GPT-3Parameters=175B, Setting=Zero-shot, Shots=02022.1060.8731.252.332.5751.431.6726.7815.912.350.547.9523.4251.538.2346.88
Cos PEParameters=3B, Method Variant=Prompt Expert, Expert Backbone=T5, Training Task=COSMOS-QA2023.0258.730.742.3651.4346.7266.6772.9225.9546.555047.0342.444.3348.13—
T0Parameters=11B, Setting=Zero-shot, Shots=02022.1058.738.352.7529.7153.6943.3369.2920.264.7351.815054.8652.1146.7915.63
PaLMParameters=540B2023.0256.5232.1645649.12559.2620.114.149.947.4724.2245.341.77—
PALMParameters=540B, Setting=Zero-shot, Shots=02022.1056.5232.1645649.12559.2620.114.149.947.4924.2245.342.1446.88
CHANNELParameters=3B, Setting=Zero-shot, Shots=02022.1052.1730.953.0116.5751.0671.6735.6711.5515.7350.4347.7925.7355.3638.0715.63
GPT-3Parameters=175B, Setting=Few-shot, Shots=32022.105031.157.158.2949.431.6735.7510.952.752.860.2733.9356.545.4856.25
T0Parameters=3B, Training=Instruction Tuning2023.0247.8332.153.2334.8653.2253.3367.2514.9458.1852.2752.0545.3344.2946.84—
T0Parameters=3B, Setting=Zero-shot, Shots=02022.1047.8341.152.7925.7152.8546.6745.5214.8458.8952.9450.2346.6444.2942.5615.63