ResearchBenchmarksInference Latency on WikiText-103, SQuAD v2, and OpenBookQAInference Latency on WikiText-103, SQuAD v2, and OpenBookQAFollow57.9Inference Latency (ms)LLMCache48.024114.687181.35248.013Dec 18, 2025Evaluation resultsMethodMethodLinksInference Latency (ms)LLMCacheModel=DistilBERTModel=DistilBERT2025.1257.9LLMCacheModel=BERT-baseModel=BERT-base2025.1291.3LLMCacheModel=GPT-2 smallModel=GPT-2 small2025.12112.5NoCacheModel=DistilBERTModel=DistilBERT2025.12123.4KV-CacheModel=GPT-2 smallModel=GPT-2 small2025.12177.3NoCacheModel=BERT-baseModel=BERT-base2025.12218.6NoCacheModel=GPT-2 smallModel=GPT-2 small2025.12304.8