MerLean-example

23 Def 17: Binary Entropy Function

Definition 261 Binary Entropy Function
#

The binary entropy function \(H_2 : \mathbb {R} \to \mathbb {R}\) is defined by

\[ H_2(\delta ) = -\delta \log _2(\delta ) - (1-\delta )\log _2(1-\delta ). \]

For \(\delta \in (0,1)\), this gives the Shannon entropy of a Bernoulli\((\delta )\) distribution measured in bits. Outside \((0,1)\), the function extends by continuity, with \(\log _2(0) = 0\) by Mathlib’s convention.

Lemma 262 Key Natural Number Inequality
#

We have the following natural number inequality:

\[ 100^{100} {\lt} 2^{50} \cdot 11^{11} \cdot 89^{89}. \]
Proof

This is verified by computation (native_decide).

Lemma 263 Key Fractional Inequality
#

We have the following inequality in \(\mathbb {R}\):

\[ \left(\frac{100}{11}\right)^{11} \cdot \left(\frac{100}{89}\right)^{89} {\lt} 2^{50}. \]
Proof

Rewriting using \(\left(\frac{a}{b}\right)^n = \frac{a^n}{b^n}\) and combining fractions, the inequality \(\frac{100^{11} \cdot 100^{89}}{11^{11} \cdot 89^{89}} {\lt} 2^{50}\) is equivalent (after clearing the positive denominator \(11^{11} \cdot 89^{89}\)) to \(100^{11} \cdot 100^{89} {\lt} 2^{50} \cdot 11^{11} \cdot 89^{89}\). By the ring identity \(100^{11} \cdot 100^{89} = 100^{100}\), this reduces to \(100^{100} {\lt} 2^{50} \cdot 11^{11} \cdot 89^{89}\), which holds by casting key_nat_ineq to \(\mathbb {R}\).

Lemma 264 \(H_2(11/100) {\lt} 1/2\)

The binary entropy function satisfies \(H_2(11/100) {\lt} 1/2\).

Proof

Unfolding the definition of \(H_2\), the goal becomes

\[ -\frac{11}{100}\log _2\! \left(\frac{11}{100}\right) - \frac{89}{100}\log _2\! \left(\frac{89}{100}\right) {\lt} \frac{1}{2}. \]

We establish auxiliary facts: \(11/100 {\gt} 0\), \(89/100 {\gt} 0\), \(\log _2(11/100) {\lt} 0\) (since \(11/100 {\lt} 1\) and the base is \({\gt} 1\)), and \(\log _2(89/100) {\lt} 0\) similarly. Using the identity \(\log _2(a/b) = \log _2 a - \log _2 b\), we rewrite:

\[ \log _2\! \left(\frac{11}{100}\right) = -\log _2\! \left(\frac{100}{11}\right), \qquad \log _2\! \left(\frac{89}{100}\right) = -\log _2\! \left(\frac{100}{89}\right). \]

After substitution and ring normalization, the goal becomes

\[ \frac{11}{100}\log _2\! \left(\frac{100}{11}\right) + \frac{89}{100}\log _2\! \left(\frac{100}{89}\right) {\lt} \frac{1}{2}. \]

We establish the key step: \(11 \cdot \log _2(100/11) + 89 \cdot \log _2(100/89) {\lt} 50\). Indeed, taking \(\log _2\) of the inequality \(\left(\frac{100}{11}\right)^{11}\cdot \left(\frac{100}{89}\right)^{89} {\lt} 2^{50}\) from key_frac_ineq, and using \(\log _2(x^n) = n\log _2 x\) and \(\log _2(2^{50}) = 50\), we obtain:

\[ 11\log _2\! \left(\frac{100}{11}\right) + 89\log _2\! \left(\frac{100}{89}\right) {\lt} 50. \]

Since \(\log _2(100/11) {\gt} 0\) and \(\log _2(100/89) {\gt} 0\), we conclude by linear arithmetic that \(\frac{11}{100}\log _2\! \left(\frac{100}{11}\right) + \frac{89}{100}\log _2\! \left(\frac{100}{89}\right) {\lt} \frac{1}{2}\).

Lemma 265 \(H_2\) equals \(\operatorname {binEntropy} / \log 2\)

For all \(\delta \in \mathbb {R}\),

\[ H_2(\delta ) = \frac{\operatorname {binEntropy}(\delta )}{\log 2}, \]

where \(\operatorname {binEntropy}\) is Mathlib’s natural-logarithm binary entropy function.

Proof

Unfolding \(H_2(\delta ) = -\delta \log _2\delta - (1-\delta )\log _2(1-\delta )\) and \(\operatorname {binEntropy}(\delta ) = -\delta \log \delta - (1-\delta )\log (1-\delta )\), and using \(\log _2 x = \log x / \log 2\) (i.e., \(\operatorname {logb}\, 2\, x = \log x / \log 2\)), together with \(\log (\cdot ^{-1}) = -\log (\cdot )\), the equality follows by ring arithmetic.

Lemma 266 Derivative of \(H_2\)

For \(x \in (0, 1)\), the binary entropy function has derivative

\[ H_2'(x) = \log _2\! \left(\frac{1-x}{x}\right). \]
Proof

Let \(x \in (0,1)\). We write \(H_2 = \operatorname {binEntropy}(\cdot )/\log 2\) by binaryEntropy_eq_binEntropy_div. Applying Mathlib’s Real.hasDerivAt_binEntropy (which gives derivative \(\log ((1-x)/x)\) for \(\operatorname {binEntropy}\) at \(x\), using \(x \neq 0\) and \(x \neq 1\)), and dividing by the constant \(\log 2\), we obtain that \(H_2\) has a derivative at \(x\). Using \(\log _2((1-x)/x) = (\log (1-x) - \log x)/\log 2 = \log ((1-x)/x)/\log 2\), the result follows.

Lemma 267 \(H_2\) is strictly increasing on \((0, 1/2)\)

The binary entropy function \(H_2\) is strictly increasing on \((0, 1/2)\): for \(0 {\lt} a {\lt} b {\lt} 1/2\), we have \(H_2(a) {\lt} H_2(b)\).

Proof

Let \(a, b \in (0, 1/2)\) with \(a {\lt} b\). Using binaryEntropy_eq_binEntropy_div, we write \(H_2(a) = \operatorname {binEntropy}(a)/\log 2\) and \(H_2(b) = \operatorname {binEntropy}(b)/\log 2\). Since \(\log 2 {\gt} 0\) (as \(1 {\lt} 2\)), it suffices to show \(\operatorname {binEntropy}(a) {\lt} \operatorname {binEntropy}(b)\). Both \(a\) and \(b\) lie in \([0, 2^{-1}]\) (verified by the hypotheses via \(\operatorname {simp}\) and linear arithmetic), and Mathlib’s Real.binEntropy_strictMonoOn asserts strict monotonicity of \(\operatorname {binEntropy}\) on \([0, 1/2]\), giving the result.

Theorem 268 \(H_2(\delta ) {\lt} 1/2\) for \(\delta \in (0, 11/100)\)

For \(\delta \in (0, 11/100)\), we have \(H_2(\delta ) {\lt} 1/2\).

Proof

Let \(\delta \) satisfy \(0 {\lt} \delta {\lt} 11/100\). We note that \(\delta {\lt} 1/2\) (by linear arithmetic from \(11/100 {\lt} 1/2\)), so \(\delta \in (0, 1/2)\). Also \(11/100 \in (0, 1/2)\). By binaryEntropy_strictMonoOn, since \(\delta {\lt} 11/100\), we have \(H_2(\delta ) {\lt} H_2(11/100)\). By binaryEntropy_at_eleven_hundredths, \(H_2(11/100) {\lt} 1/2\). Chaining these inequalities gives \(H_2(\delta ) {\lt} 1/2\).

Definition 269 Hamming Ball
#

The Hamming ball of radius \(r \in \mathbb {R}\) centered at the origin in \(\mathbb {F}_2^n\) is the finset

\[ B_0(r) = \{ v \in \mathbb {F}_2^n : \operatorname {wt}(v) \leq r \} , \]

where \(\operatorname {wt}(v)\) denotes the Hamming weight of \(v\). When \(r {\lt} 0\), the ball is empty.

Theorem 270 Membership in the Hamming Ball

For \(v : \operatorname {Fin} n \to \mathbb {F}_2\) and \(r \in \mathbb {R}\),

\[ v \in B_0(r) \iff \operatorname {wt}(v) \leq r. \]
Proof

This follows immediately by unfolding the definition of \(B_0(r)\) and applying simp.

Lemma 271 Hamming Ball is Empty for Negative Radius

For \(r {\lt} 0\), \(B_0(r) = \emptyset \).

Proof

By extensionality, it suffices to show that no \(v\) belongs to \(B_0(r)\). Suppose \(v \in B_0(r)\); then by simp on the definition, \(\operatorname {wt}(v) \leq r {\lt} 0\). But \(\operatorname {wt}(v) \geq 0\) since it is a natural number cast to \(\mathbb {R}\), a contradiction by linear arithmetic. The other direction is trivial since \(\emptyset \) has no members.

Lemma 272 Hamming Ball is Nonempty for Nonnegative Radius

For \(r \geq 0\), \(B_0(r)\) is nonempty.

Proof

We exhibit the zero vector \(0 \in \mathbb {F}_2^n\) as a witness. By mem_hammingBall, it suffices to show \(\operatorname {wt}(0) \leq r\). Using simp with the definitions of hammingWeight, hammingNorm, and hammingDist_self, we get \(\operatorname {wt}(0) = 0\), and \(0 \leq r\) holds by assumption.

Lemma 273 Bernoulli Weight Monotonicity
#

Let \(0 {\lt} \delta \leq 1/2\) and \(a \leq k \leq n\). Then

\[ \delta ^k (1-\delta )^{n-k} \leq \delta ^a (1-\delta )^{n-a}. \]
Proof

We establish \(0 {\lt} 1 - \delta \) (by linear arithmetic from \(\delta \leq 1/2\)) and \(\delta \leq 1 - \delta \) (since \(\delta \leq 1/2\)). Writing \(k = a + (k - a)\) and \(n - a = (k-a) + (n-k)\), and rewriting the powers accordingly, the left-hand side becomes

\[ \delta ^a \cdot \delta ^{k-a} \cdot (1-\delta )^{n-k} = \delta ^a \cdot (1-\delta )^{n-k} \cdot \delta ^{k-a}. \]

Since \(\delta \leq 1 - \delta \) and \(\delta {\gt} 0\), we have \(\delta ^{k-a} \leq (1-\delta )^{k-a}\) by pow_le_pow_left. Applying mul_le_mul_of_nonneg_left twice (first using \((1-\delta )^{n-k} {\gt} 0\), then \(\delta ^a {\gt} 0\)), we obtain

\[ \delta ^a \cdot (1-\delta )^{n-k} \cdot \delta ^{k-a} \leq \delta ^a \cdot (1-\delta )^{n-k} \cdot (1-\delta )^{k-a} = \delta ^a \cdot (1-\delta )^{n-a}, \]

completing the proof by ring arithmetic.

Lemma 274 Bernoulli Weight Sum

For \(0 {\lt} \delta \leq 1/2\),

\[ \sum _{v : \mathbb {F}_2^n} \delta ^{\operatorname {wt}(v)} (1-\delta )^{n - \operatorname {wt}(v)} = 1. \]
Proof

We note that every element of \(\mathbb {F}_2\) is either \(0\) or \(1\) (by decide). We define an equivalence \(e : (\operatorname {Fin} n \to \mathbb {F}_2) \simeq \operatorname {Finset}(\operatorname {Fin} n)\) by mapping \(v\) to its support \(\{ i \in \operatorname {Fin} n : v_i \neq 0\} \), with inverse sending a subset \(s\) to the indicator function \(\mathbf{1}_s\). The left inverse is verified case-by-case on \(\mathbb {F}_2\), and the right inverse by a boolean case split. Under this equivalence, the cardinality of \(e(v)\) equals \(\operatorname {wt}(v)\) by definition. Applying Mathlib’s Fin.sum_pow_mul_eq_add_pow, we get

\[ \sum _{s : \operatorname {Finset}(\operatorname {Fin} n)} \delta ^{|s|}(1-\delta )^{n-|s|} = (\delta + (1-\delta ))^n = 1^n = 1. \]

Transporting along \(e\) via Equiv.sum_comp and a congruence argument gives the result.

Lemma 275 Entropy Logarithm Inequality

Let \(0 {\lt} \delta \leq 1/2\), \(k \leq n\), and \((k : \mathbb {R}) + 1 \leq \delta \cdot n\). Then

\[ -(k \log _2 \delta + (n - k)\log _2(1-\delta )) \leq H_2(\delta ) \cdot n. \]
Proof

Unfolding \(H_2(\delta ) = -\delta \log _2\delta - (1-\delta )\log _2(1-\delta )\), we establish that \(\delta n - k {\gt} 0\) (by linear arithmetic from \(k + 1 \leq \delta n\)) and that \(\log _2 \delta \leq \log _2(1-\delta )\) (by strict monotonicity of \(\log _2\) on \((0,\infty )\), applied at points \(\delta {\lt} 1 - \delta \) since \(\delta \leq 1/2\)). Casting \(n - k\) appropriately, the inequality then follows by nonlinear arithmetic (nlinarith), using \((\delta n - k) \cdot (\log _2(1-\delta ) - \log _2\delta ) \geq 0\).

Let \(n \geq 1\), \(0 {\lt} \delta \leq 1/2\), and \(k + 1 \leq \delta n\). Then

\[ |B_0(k)| \leq 2^{H_2(\delta ) \cdot n}. \]
Proof

We establish \(0 {\lt} 1 - \delta \) by linear arithmetic. We show \(k \leq n\): otherwise \(k {\gt} n\), so \(\delta n {\lt} n\), but \(k + 1 \leq \delta n {\lt} n \leq k\), a contradiction by linear arithmetic. Let \(p = \delta ^k(1-\delta )^{n-k} {\gt} 0\) (by positivity).

Step 1: \(|B_0(k)| \cdot \delta ^k(1-\delta )^{n-k} \leq 1\). We compute:

\[ |B_0(k)| \cdot p = \sum _{v \in B_0(k)} p \leq \sum _{v \in B_0(k)} \delta ^{\operatorname {wt}(v)}(1-\delta )^{n-\operatorname {wt}(v)} \leq \sum _{v : \mathbb {F}_2^n} \delta ^{\operatorname {wt}(v)}(1-\delta )^{n-\operatorname {wt}(v)} = 1. \]

The first inequality uses bernoulli_weight_mono: for \(v \in B_0(k)\), \(\operatorname {wt}(v) \leq k\), so \(\delta ^k(1-\delta )^{n-k} \leq \delta ^{\operatorname {wt}(v)}(1-\delta )^{n-\operatorname {wt}(v)}\). The second uses Finset.sum_le_univ_sum_of_nonneg (terms are nonneg). The final equality is bernoulli_weight_sum.

Step 2: \(p^{-1} \leq 2^{H_2(\delta ) n}\). This is equivalent (by Real.logb_le_iff_le_rpow) to \(\log _2(p^{-1}) \leq H_2(\delta ) n\). Using \(\log _2(p^{-1}) = -\log _2 p\) and \(\log _2(\delta ^k (1-\delta )^{n-k}) = k\log _2\delta + (n-k)\log _2(1-\delta )\), this becomes \(-(k\log _2\delta + (n-k)\log _2(1-\delta )) \leq H_2(\delta )n\), which holds by entropy_logb_ineq.

Conclusion: From Step 1, \(|B_0(k)| \leq p^{-1}\), and from Step 2, \(p^{-1} \leq 2^{H_2(\delta )n}\), so by linear arithmetic \(|B_0(k)| \leq 2^{H_2(\delta )n}\).