# Kernel-Based ReLU Approximation for Homomorphic Encryption-Compatible Privacy-preserving Deep Learning Models Source: [Kernel-Based ReLU Approximation for Homomorphic Encryption-Compatible Privacy-preserving Deep Learning Models]() Collection index: [llms.txt]() Data updated: 2026-05-26T05:00:13.075Z Authors: Dimitrios Sygletos, Dimitra Papatsaroucha, Marios Choudetsanakis, Ilias Politis, Evangelos K. Markakis Published: 2026-05-22T13:54:37.000Z ## Abstract As privacy concerns in AI technologies continue to grow, Homomorphic Encryption (HE) offers a way to perform computations on encrypted data without the need of decryption during operations. However, HE is limited to addition and multiplication, making non-linear functions incompatible in their original form. This limitation has become more critical with the widespread use of Large Language Models (LLMs), where the non-linearity of activation functions such as the Rectified Linear Unit (ReLU) poses challenges for deployment in privacy-preserving Natural Language Processing (NLP) settings. This paper proposes a kernel-based approximation of ReLU, enabling its use within HE-constrained settings and thus contributing a critical step toward supporting privacy-preserving LLMs. A smooth kernel-based function, mimicking ReLU, is approximated using a second-degree polynomial, inspired by Jackson's theorem, to achieve low multiplicative depth. The proposed method is trained and assessed directly on token embeddings from pre-trained LLMs and evaluated in various scenarios, from simulated and tokenized data to deep learning and transformer models. Results show improved approximation fidelity, supporting the method's suitability for secure and privacy-preserving inference in various tasks. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Image Classification on MNIST CKKS (encrypted)](): Image Classification; MNIST CKKS (encrypted); Total Exp. Time (s): 1864.19. - [ReLU Approximation on SST-2 RoBERTa tokenized vectors (test)](): ReLU approximation; SST-2 RoBERTa tokenized vectors (test); MSE: 0.002. - [ReLU Approximation on SST-2 DistilBERT tokenized vectors (test)](): ReLU approximation; SST-2 DistilBERT tokenized vectors (test); MSE: 0.002. - [Image Classification on MNIST (Plaintext and Encrypted Accuracy)](): Image Classification; MNIST; Plaintext Accuracy: 97.3. - [Image Classification on CIFAR-100 (ACC, NLL, ECE, ESS)](): Image Classification; CIFAR-100; Accuracy: 38.5. - [Image Classification on CIFAR-10 (Accuracy)](): Image Classification; CIFAR-10; Accuracy: 64.7. ## Original sources and implementation [Original paper]() [PDF]()