# MIMONets: Multiple-Input-Multiple-Output Neural Networks Exploiting Computation in Superposition

Source: [MIMONets: Multiple-Input-Multiple-Output Neural Networks Exploiting Computation in Superposition](<https://www.sota2.com/research/paper/68fa715a1e209131cea50c21>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-04T04:40:31.319Z

Authors: Nicolas Menet, Michael Hersche, Geethan Karunaratne, Luca Benini, Abu Sebastian, Abbas Rahimi (1) \_\_INSTITUTION\_6\_\_ IBM Research - Zurich, (2) ETH Zurich)

Published: 2023-12-05T15:25:45.000Z

## Abstract

With the advent of deep learning, progressively larger neural networks have been designed to solve complex tasks. We take advantage of these capacity-rich models to lower the cost of inference by exploiting computation in superposition. To reduce the computational burden per input, we propose Multiple-Input-Multiple-Output Neural Networks (MIMONets) capable of handling many inputs at once. MIMONets augment various deep neural network architectures with variable binding mechanisms to represent an arbitrary number of inputs in a compositional data structure via fixed-width distributed representations. Accordingly, MIMONets adapt nonlinear neural transformations to process the data structure holistically, leading to a speedup nearly proportional to the number of superposed input items in the data structure. After processing in superposition, an unbinding mechanism recovers each transformed input of interest. MIMONets also provide a dynamic trade-off between accuracy and throughput by an instantaneous on-demand switching between a set of accuracy-throughput operating points, yet within a single set of fixed parameters. We apply the concept of MIMONets to both CNN and Transformer architectures resulting in MIMOConv and MIMOFormer, respectively. Empirical evaluations show that MIMOConv achieves about 2-4 x speedup at an accuracy delta within \[+0.68, -3.18\]% compared to WideResNet CNNs on CIFAR10 and CIFAR100. Similarly, MIMOFormer can handle 2-4 inputs at once while maintaining a high average accuracy within a \[-1.07, -3.43\]% delta on the long range arena benchmark. Finally, we provide mathematical bounds on the interference between superposition channels in MIMOFormer. Our code is available at https://github.com/IBM/multiple-input-multiple-output-nets.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Associative Recall on Synthetic sequence modeling Associative recall](<https://www.sota2.com/research/sota/associative-recall-on-synthetic-sequence-modeling-associative-recall.md>): Associative Recall; Synthetic sequence modeling Associative recall; Accuracy: 96.52.

- [Induction head on Synthetic sequence modeling Induction head](<https://www.sota2.com/research/sota/induction-head-on-synthetic-sequence-modeling-induction-head.md>): Induction head; Synthetic sequence modeling Induction head; Accuracy: 99.4.

- [Image Classification on CIFAR10 (test) (Test Accuracy)](<https://www.sota2.com/research/sota/image-classification-on-cifar10-test-test-accuracy.md>): Image Classification; CIFAR10 (test); Test Accuracy: 97.49.

- [Long-range Sequence Modeling on Long Range Arena (LRA) (test)](<https://www.sota2.com/research/sota/long-range-sequence-modeling-on-long-range-arena-lra-test.md>): Long-range sequence modeling; Long Range Arena (LRA) (test); Accuracy (Text): 65.39.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2312.02829>)

[PDF](<https://arxiv.org/pdf/2312.02829.pdf>)

[Implementation](<https://github.com/IBM/multiple-input-multiple-output-nets>)
