Patent Claim Validity Classification on ACE-40k (test)
95.05AccuracyACE Hybrid
Evaluation Results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| ACE HybridProcessing Stage=Stage 1 + 2, Components=Gatekeeper + Stage 2 Expert LLM2026.04 | 95.05 | 94.95 | — | 1.5 | 1,247 | |
| Llama-3-70B SFT w/ CoPTTraining Strategy=CoPT, Mode=Expert LLM Only, Backbone=Llama-3-70B2026.04 | 94.8 | 94.76 | — | 6.88 | 5,733 | |
| Llama-3-70B SFTEvaluation Protocol=Zero-shot, Training Strategy=SFT, Backbone=Llama-3-70B2026.04 | 94.6 | 94.63 | — | 11.85 | 9,875 | |
| PatentBERT Fine-tunedTraining Strategy=Fine-tuned2026.04 | 91.65 | 91.58 | 97.16 | 0.12 | 100 | |
| ACE GatekeeperProcessing Stage=Stage 1 Only, Backbone=PatentBERT-based encoder2026.04 | 91.65 | 91.58 | 97.16 | 0.12 | 100 | |
| PatentBERTModel Category=Domain-Specific Experts2026.04 | 77.25 | 76.59 | 51.16 | 0.12 | 100 | |
| Vanilla LongformerModel Category=Sanity Check & General Baselines2026.04 | 75.52 | 73.11 | 47.26 | 0.42 | 350 | |
| TF-IDF + Logistic RegressionMethod=Lexical Baseline, Classifier=Logistic Regression2026.04 | 73.38 | 72.23 | 83 | 0.01 | 1 | |
| Llama-3-70B BaseEvaluation Protocol=Zero-shot, Backbone=Llama-3-70B2026.04 | 70.1 | 67.75 | — | 7.31 | 6,092 | |
| Legal-BERTModel Category=Domain-Specific Experts2026.04 | 69.7 | 68.86 | 46.3 | 0.12 | 100 | |
| Frozen BERT + Logistic RegressionTraining Strategy=Frozen, Classifier=Logistic Regression2026.04 | 69.27 | 68.12 | 66.2 | 0.08 | 67 | |
| Llama-3-70B BaseEvaluation Protocol=Few-shot, Backbone=Llama-3-70B2026.04 | 60 | 34.85 | — | 3.32 | 2,767 | |
| Vanilla BERTModel Category=Sanity Check & General Baselines2026.04 | 49.6 | 66.25 | 50.93 | 0.12 | 100 |