# Doubly Sparse: Sparse Mixture of Sparse Experts for Efficient Softmax Inference

Source: [Doubly Sparse: Sparse Mixture of Sparse Experts for Efficient Softmax Inference](<https://www.sota2.com/research/paper/68fa65fc1e209131cea0db93>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T21:40:30.056Z

Authors: Shun Liao, Ting Chen, Tian Lin, Denny Zhou, Chong Wang

Published: 2019-01-30T04:22:03.000Z

## Abstract

Computations for the softmax function are significantly expensive when the number of output classes is large. In this paper, we present a novel softmax inference speedup method, Doubly Sparse Softmax (DS-Softmax), that leverages sparse mixture of sparse experts to efficiently retrieve top-k classes. Different from most existing methods that require and approximate a fixed softmax, our method is learning-based and can adapt softmax weights for a better inference speedup. In particular, our method learns a two-level hierarchy which divides entire output class space into several partially overlapping experts. Each expert is sparse and only contains a subset of output classes. To find top-k classes, a sparse mixture enables us to find the most probable expert quickly, and the sparse expert enables us to search within a small-scale softmax. We empirically conduct evaluation on several real-world tasks, including neural machine translation, language modeling and image classification, and demonstrate that significant computation reductions can be achieved at no performance loss.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/1901.10668>)

[PDF](<https://arxiv.org/pdf/1901.10668.pdf>)
