# The Upper Bound on Knots in Neural Networks

Source: [The Upper Bound on Knots in Neural Networks](<https://www.sota2.com/research/paper/68fa63031e209131ce9fc585>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T20:31:19.733Z

Authors: Kevin K. Chen

Published: 2016-11-29T01:07:58.000Z

## Abstract

Neural networks with rectified linear unit activations are essentially multivariate linear splines. As such, one of many ways to measure the "complexity" or "expressivity" of a neural network is to count the number of knots in the spline model. We study the number of knots in fully-connected feedforward neural networks with rectified linear unit activation functions. We intentionally keep the neural networks very simple, so as to make theoretical analyses more approachable. An induction on the number of layers $l$ reveals a tight upper bound on the number of knots in $\\mathbb{R} \\to \\mathbb{R}^p$ deep neural networks. With $n\_i \\gg 1$ neurons in layer $i = 1, \\dots, l$, the upper bound is approximately $n\_1 \\dots n\_l$. We then show that the exact upper bound is tight, and we demonstrate the upper bound with an example. The purpose of these analyses is to pave a path for understanding the behavior of general $\\mathbb{R}^q \\to \\mathbb{R}^p$ neural networks.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/1611.09448>)

[PDF](<https://arxiv.org/pdf/1611.09448.pdf>)
