# Occam's Ghost

Source: [Occam's Ghost](<https://www.sota2.com/research/paper/68fa57081e209131ce9a20c0>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T14:31:44.072Z

Authors: Peter K\\"ovesarki

Published: 2020-06-15T20:25:09.000Z

## Abstract

This article applies the principle of Occam's Razor to non-parametric model building of statistical data, by finding a model with the minimal number of bits, leading to an exceptionally effective regularization method for probability density estimators. The idea comes from the fact that likelihood maximization also minimizes the number of bits required to encode a dataset. However, traditional methods overlook that the optimization of model parameters may also inadvertently play the part in encoding data points. The article shows how to extend the bit counting to the model parameters as well, providing the first true measure of complexity for parametric models. Minimizing the total bit requirement of a model of a dataset favors smaller derivatives, smoother probability density function estimates and most importantly, a phase space with fewer relevant parameters. In fact, it is able prune parameters and detect features with small probability at the same time. It is also shown, how it can be applied to any smooth, non-parametric probability density estimator.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2006.09813>)

[PDF](<https://arxiv.org/pdf/2006.09813.pdf>)
