# Combinatorial Entropies and Statistics

Source: [Combinatorial Entropies and Statistics](<https://www.sota2.com/research/paper/68fa2eed1e209131ce867358>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-28T13:53:30.021Z

Authors: Robert K. Niven

Published: 2009-02-18T03:13:54.000Z

## Abstract

We examine the {combinatorial} or {probabilistic} definition ("Boltzmann's principle") of the entropy or cross-entropy function $H \\propto \\ln \\mathbb{W}$ or $D \\propto - \\ln \\mathbb{P}$, where $\\mathbb{W}$ is the statistical weight and $\\mathbb{P}$ the probability of a given realization of a system. Extremisation of $H$ or $D$, subject to any constraints, thus selects the "most probable" (MaxProb) realization. If the system is multinomial, $D$ converges asymptotically (for number of entities $N \\back \\to \\back \\infty$) to the Kullback-Leibler cross-entropy $D\_{KL}$; for equiprobable categories in a system, $H$ converges to the Shannon entropy $H\_{Sh}$. However, in many cases $\\mathbb{W}$ or $\\mathbb{P}$ is not multinomial and/or does not satisfy an asymptotic limit. Such systems cannot meaningfully be analysed with $D\_{KL}$ or $H\_{Sh}$, but can be analysed directly by MaxProb. This study reviews several examples, including (a) non-asymptotic systems; (b) systems with indistinguishable entities (quantum statistics); (c) systems with indistinguishable categories; (d) systems represented by urn models, such as "neither independent nor identically distributed" (ninid) sampling; and (e) systems representable in graphical form, such as decision trees and networks. Boltzmann's combinatorial definition of entropy is shown to be of greater importance for {"probabilistic inference"} than the axiomatic definition used in information theory.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/0902.3038>)

[PDF](<https://arxiv.org/pdf/0902.3038.pdf>)
