# LookupFFN: Making Transformers Compute-lite for CPU inference

Source: [LookupFFN: Making Transformers Compute-lite for CPU inference](<https://www.sota2.com/research/paper/68fa727e1e209131cea56bf3>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-27T06:31:06.454Z

Authors: Zhanpeng Zeng, Michael Davies, Pranav Pulijala, Karthikeyan Sankaralingam, Vikas Singh

Published: 2024-03-12T00:26:16.000Z

## Abstract

While GPU clusters are the de facto choice for training large deep neural network (DNN) models today, several reasons including ease of workflow, security and cost have led to efforts investigating whether CPUs may be viable for inference in routine use in many sectors of the industry. But the imbalance between the compute capabilities of GPUs and CPUs is huge. Motivated by these considerations, we study a module which is a workhorse within modern DNN architectures, GEMM based Feed Forward Networks (FFNs), and assess the extent to which it can be made compute- (or FLOP-) lite. Specifically, we propose an alternative formulation (we call it LookupFFN) to GEMM based FFNs inspired by the recent studies of using Locality Sensitive Hashing (LSH) to approximate FFNs. Our formulation recasts most essential operations as a memory look-up, leveraging the trade-off between the two resources on any platform: compute and memory (since CPUs offer it in abundance). For RoBERTa language model pretraining, our formulation achieves similar performance compared to GEMM based FFNs, while dramatically reducing the required FLOP. Our development is complemented with a detailed hardware profiling of strategies that will maximize efficiency -- not just on contemporary hardware but on products that will be offered in the near/medium term future. Code is avaiable at \\url{https://github.com/mlpen/LookupFFN}.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Language Modeling on C4 (Log-PPL)](<https://www.sota2.com/research/sota/language-modeling-on-c4-log-ppl.md>): Language Modeling; C4; Log-PPL: 2.865.

- [Common Sense Reasoning on Hellaswag](<https://www.sota2.com/research/sota/common-sense-reasoning-on-hellaswag.md>): Common Sense Reasoning; HellaSwag; Accuracy: 45.96.

- [Question Answering on ARC Challenge](<https://www.sota2.com/research/sota/question-answering-on-arc-challenge.md>): Question Answering; ARC Challenge; Accuracy: 60.68.

- [Language Modeling on WikiText-2](<https://www.sota2.com/research/sota/language-modeling-on-wikitext-2.md>): Language Modeling; WikiText-2; Perplexity (PPL): not recorded.

- [Language Modeling on PTB](<https://www.sota2.com/research/sota/language-modeling-on-ptb.md>): Language Modeling; PTB; Perplexity: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2403.07221>)

[PDF](<https://arxiv.org/pdf/2403.07221.pdf>)