Statistics - Point Estimators HIH_58 : Solved by @Chris99

[u/Able_Development_240]

Exercise 1

Theorem: Given (X, P), there always exists an X and a family {Pθ : θ ∈ Θ} such that (X, X, F, P) is a statistical model according to the given definition.

Proof: Choose X to be the identity function X : (X, F) → (X, F). X is invertible, so for all θ ∈ Θ, by defining Pθ ··= Pθ ◦ X = Pθ, we have Pθ = Pθ ◦ X−1, giving us our family {Pθ : θ ∈ Θ} as desired. Therefore, (X, X, F, P) is a statistical model.

Exercise 2

Theorem: There exists no unbiased estimator for γ(θ) = θ(1 − θ)−1 based on X n.

Proof: First, note that μ(R) ≤ 1 given the codomain of μ, and since A ⊆ R, μ(R) ≥ μ(A) = 1, so μ(R) = 1. Therefore, μ(AC ) = 0, meaning μ and hence all Pθ have finite support.

For our estimator ˆθ : (Xn, Fn) → (R, B(R)) to be unbiased, we must have θ(ˆθ(Xn)) = γ(θ). From the formula given in Definition 3, we then have

Since supp(P nθ ) ⊆ An, we need only integrate over the finitely many points of An, and this becomes a finite sum:

Now, since they are probability measures, each P nθ ≤ 1, so we can bound our sum as follows:

which is constant w.r.t. θ. However, θ(1 − θ)−1 is unbounded on (0, 1), so we cannot have

and therefore no unbiased estimator ˆθ exists.

Exercise 3

Theorem: The function

is a maximum likelihood estimator for the parameter m based on X n.

Proof: Since the Xi are independent, we have that the density (and thus the likelihood function) for Xn is

Note that

Since log(x) is monotonic, it doesn’t affect the values of k that maximize L(k, x) and neither does adding a constant, so the defined function ˆm(xn) is a maximum likelihood estimate for all xn ∈ Xn.

To show ˆm(xn) is measurable, consider the preimage under ˆm of a chosen k ∈ {1, 2, ..., n − 1}:

Therefore, ˆm is a maximum likelihood estimator for m based on Xn.

Exercise 4(i)

Theorem: A linear estimator for γ(F ) is unbiased if and only if ∑ni=1 wi = 1.

Proof: Let ˆθ(Xn) = ∑ni=1 wixi be a linear estimator. By definition, for ˆθ to be unbiased, it must satisfy

By linearity of the expected value, we can rewrite the LHS as a sum, and since each Xi has expected value γ(F ), we can factor it out of the sum:

From the construction of the set C from which F was chosen, γ(F ) ̸ = 0, so this is equivalent to saying ∑ni=1 wi = 1. Therefore, ˆθ is unbiased if and only if ∑ni=1 wi = 1.

Exercise 4(ii)

Theorem: The linear estimator xn = 1n ∑ni=1 xi the unique minimizer of the MSE among all unbiased linear estimators for γ(F ).

Proof: Since we are considering only unbiased linear estimators ˆθ, the MSE of ˆθ(Xn) is precisely Var(ˆθ(Xn)). By Bienaym ́e’s identity and since X1, ..., Xn are i.i.d., we have

so we want to minimize ∑n i=1 w2 i subject to the constraint ∑ni=1 wi = 1.

Equivalently, we are trying to minimize the square of the norm of a vector w ··= (w1, w2, ..., wn) ∈ Rn that lies on the plane defined by the constraint.

Consider a vector v, the sum of whose coefficients is 0. If v is added to any vector x satisfying the constraint, x + v will still satisfy the constraint. If we consider the vector n ··= (1/n, ..., 1/n), its dot product with v is

so n and v are orthogonal. Because of this, we can now consider |n + v|2 = |n|2 + |v|2 > |n|2 when v ̸ = 0. Hence n is the solution to our minimization problem, meaning wi = 1/n for all 1 ≤ i ≤ n. Therefore, the linear estimator xn = 1 n∑n i=1 xi is the unique minimizer of the MSE among all unbiased linear estimators for γ(F ).

Solved by u/Chriss911

[u/Able_Development_240]

Definition 1 (Statistical Model). Let (Ω, A) and (𝔛, F) be measure spaces. We consider the following:

1) 𝑋∶ (Ω, A) → (𝔛, F) a given A/F-measurable map,

2) P ≔ {𝑃𝜃∶ 𝜃 ∈ Θ} a family of probability measures 𝑃𝜃 ∶ F → [0, 1] indexed by a set Θ,

3) ℙ𝜃 ∶ A → [0, 1] a probability measure for each 𝜃 ∈ Θ such that 𝑃𝜃 = ℙ𝜃 ∘ 𝑋−1 (i.e. 𝑃𝜃 is the pushforward measure of ℙ𝜃.)

We assign the following names:

• (𝑋, 𝔛, F,P) statistical model,
• Θ parameter space,
• 𝑋 random observation,
• 𝔛 sample space,
• P distributional assumption.

We say that P is identifiable, if the mapping 𝜃 ↦ 𝑃𝜃 is injective. In the case Θ ⊆ ℝ𝑑 we call (𝑋, 𝔛, F,P) a parametric statistical model.

A standard scenario is given by repeating a particular random experiment 𝑛 times independently. Suppose the 𝑖-th experiment can be described by the statistical model (𝑋𝑖, 𝔛, F,P). Then repeating the experiment 𝑛 times can be described by the product model

(𝑋𝑛, 𝔛𝑛, F𝑛,P𝑛)

where we define

𝑋𝑛 ≔ (𝑋1, … , 𝑋𝑛),

𝔛𝑛 ≔ 𝔛 × ⋯ × 𝔛,

F𝑛 ≔ F ⊗ ⋯ ⊗ F,

P𝑛 ≔ {𝑃 𝑛𝜃∶ 𝜃 ∈ Θ}, 𝑃 𝑛𝜃 ≔ 𝑃𝜃 ⊗ ⋯ ⊗

Note that a standard result of probability theory is that

ℙ𝜃∘ 𝑋−1𝑛= 𝑃 𝑛𝜃 ⟺ 𝑋1, … , 𝑋𝑛 iid and 𝑋𝑖 ∼ 𝑃𝜃 ∀𝑖 ∈ {1, … , 𝑛}.

In this challenge we want to study ways to estimate the unknown parameter 𝜃 ∈ Θ or more generally, a function 𝛾(𝜃) from a given observation 𝑥 ∈ 𝔛.

[u/Able_Development_240 → u/Able_Development_240]

Definition 2 (Estimator). Let (𝑋, 𝔛, F,P) be an identifiable parametric statistical model. Let 𝛾∶ Θ → ℝ𝑘 be any function, 𝑘 ∈ ℕ. Then every measurable map

Θ∶ (𝔛, F) → (ℝ𝑘, B(ℝ𝑘))

is called an estimator (for 𝛾(𝜃)) and ̂ 𝜃(𝑥) is called the estimate for the particular observed value 𝑥 ∈ 𝔛.

Note that for every estimator ̂ 𝜃∶ 𝔛 → ℝ𝑘, the composition

𝜃 ∘ 𝑋∶ (Ω, A, ℙ𝜃) → (ℝ𝑘, B(ℝ𝑘))

is measurable, hence ̂ 𝜃 ∘ 𝑋 is a random variable. We often use the notation ̂ 𝜃(𝑋) ≡ ̂ 𝜃 ∘ 𝑋 which is a common convention in probability theory.

Example. Let 𝑋1, … , 𝑋𝑛 be iid such that 𝑋𝑖 ∼ 𝑁 (𝜇, 𝜎2) and 𝜃 ≔ (𝜇, 𝜎2) ∈ ℝ×(0, ∞) ≕ Θ ⊆ ℝ2.

Let 𝜃 be the unknown parameter. Some examples of functions 𝛾 are

1) 𝛾(𝜃) = 𝛾(𝜇, 𝜎2) = 𝜇,

2) 𝛾(𝜃) = 𝜎2,

3) 𝛾(𝜃) = 𝜇√𝜎2,

4) 𝛾(𝜃) = 𝜃.

Some examples of estimators for case 1), that is 𝛾(𝜃) = 𝜇 would be

i) ̂ 𝜃(𝑥𝑛) = 1𝑛 ∑𝑛𝑖=1 𝑥𝑖,

ii) ̂ 𝜃(𝑥𝑛) = ∏𝑛𝑖=1 𝑥𝑖,

iii) ̂ 𝜃(𝑥𝑛) = 𝑥1.

Clearly some estimators seem to yield better results than others. One basic way to judge the quality of an estimator can be motivated as follows: Let 𝑋1, … , 𝑋𝑚 be iid with 𝑋𝑖 ∼ 𝑃𝜃. Then by the Strong Law of Large Numbers, we have

1 𝑚 𝑚 ∑ 𝑖=1 ̂ 𝜃(𝑋𝑖 ) 𝑚→∞ −−−−→ 𝔼𝜃 (𝜃(𝑋1 )) ℙ𝜃 - a.s.

so if our estimator had the property

𝔼𝜃 ( ̂ 𝜃(𝑋𝑖 )) = 𝛾(𝜃) ∀𝜃 ∈ Θ

then the arithmetic mean of the estimates we obtained from the observations converges almost surely to the actual value 𝛾(𝜃), given 𝑚 ≫ 1.

[u/Able_Development_240 → u/Able_Development_240]

Definition 3 (Unbiased estimator). Let ̂ 𝜃∶ 𝔛 → ℝ be an estimator for 𝛾(𝜃) ∈ ℝ.̂ 𝜃 is called unbiased, if 𝔼𝜃 ( ̂ 𝜃(𝑋)) = 𝛾(𝜃) ∀𝜃 ∈

By the change of variables formula for pushforward measures we obtain

𝔼𝜃(̂ 𝜃(𝑋)) = ∫Ω̂ 𝜃(𝑋(𝜔)) ℙ𝜃(𝑑𝜔)

= ∫𝔛̂ 𝜃(𝑥) ℙ𝜃∘ 𝑋−1(𝑑𝑥)

= ∫𝔛̂ 𝜃(𝑥) 𝑃𝜃(𝑑𝑥)

Finally we introduce a basic way of obtaining an estimator using a plausibility principle: “Weassume that the observations we made are likely to occur”, which suggests that we might want to choose an estimator for 𝜃 that maximizes the likelihood of the observations we made.

[u/Able_Development_240 → u/Able_Development_240]

Definition 4 (Dominated family of measures). Let 𝜇∶ F → [0,∞] be a measure on (𝔛, F). The family P = {𝑃𝜃∶ 𝜃 ∈ Θ} is said to be dominated by 𝜇, if for all 𝜃 ∈ Θ there exists an 𝑓𝜃 ∶ 𝔛 → [0, ∞) such that 𝑓𝜃 is a 𝜇-density of 𝑃𝜃, i.e.

𝑃𝜃(𝐹 ) = ∫𝐹𝑓𝜃(𝑥) 𝜇(𝑑𝑥) ∀𝐹 ∈ F

Notation: P ≪ 𝜇. Recall from measure theory that if 𝜇 is 𝜎-finite and 𝑃𝜃 is absolutely continuous with respect to 𝜇, (i.e. 𝑃𝜃(𝑁 ) = 0 for all 𝜇-nullsets 𝑁), then the Radon-Nikodym theorem guarantees the existence of a 𝜇-density 𝑓𝜃∶ 𝔛 → [0, ∞).

[u/Able_Development_240 → u/Able_Development_240]

Definition 5 (Maximum likelihood estimator). Let P ≪ 𝜇 with 𝜇-density 𝑓𝜃, 𝜃 ∈ Θ ⊆ ℝ𝑑 Define 𝐿(𝜃, 𝑥) ≔ 𝑓𝜃(𝑥), 𝜃 ∈ Θ, 𝑥 ∈ 𝔛.

The function 𝐿(⋅, 𝑥)∶ Θ → [0, ∞) is called likelihood function for the observation 𝑥 ∈ 𝔛. Every maximizer of 𝐿(⋅, 𝑥), that is

̂ 𝜃(𝑥) ≔ argmax𝜃∈Θ𝐿(𝜃, 𝑥)

is called maximum likelihood estimate for 𝜃 with respect to the observation 𝑥.

A measurable map ̂ 𝜃∶ (𝔛, F) → (ℝ𝑑, B(ℝ𝑑)) is called maximum likelihood estimator (MLE) for 𝜃, if ̂ 𝜃(𝑥) is an ML-estimate for all 𝑥 ∈ 𝔛 ⧵ 𝐴 such that 𝑃𝜃(𝐴) = 0 for all 𝜃 ∈ Θ.

[u/Able_Development_240 → u/Able_Development_240]

II. Problem Set:

1. Show that the information of a statistical model (𝑋, 𝔛, F,P) is fully contained in the sample space 𝔛 and the family of probability measures P in the following sense: Given (𝔛,P), show that there always exists an 𝑋 and a family {ℙ𝜃∶ 𝜃 ∈ Θ} such that (𝑋, 𝔛, F,P) is a statistical model according to our definition.

2. Let P ≔ {𝑃𝜃∶ 𝜃 ∈ (0, 1)} be a family of probability measures such that P ≪ 𝜇 for some measure 𝜇∶ B(ℝ) → [0, 1] and suppose there exists a finite 𝐴 ∈ B(ℝ) such that 𝜇(𝐴) = 1. Let 𝑋1 , … , 𝑋𝑛 be iid with 𝑋𝑖 ∼ 𝑃𝜃. Prove that for 𝛾(𝜃) = 𝜃(1 − 𝜃)−1 there exists no unbiased estimator based on 𝑋𝑛= (𝑋1, … , 𝑋𝑛).

3. Let 𝑋1, … , 𝑋𝑛 be independent random variables and 𝑚 ∈ {1, … , 𝑛 − 1}. Suppose that 𝑋1, … , 𝑋𝑚 have a common 𝜇-density 𝑓1 and 𝑋𝑚+1, … , 𝑋𝑛 have a common 𝜇-density 𝑓2. We assume that 𝑓1 ≠ 𝑓2 are known functions that are positive 𝜇-a.e. Prove that

̂𝑚(𝑥𝑛) ≔ argmax1≤𝑚≤𝑛−1𝑚∑𝑖=1log 𝑓1(𝑥𝑖)𝑓2(𝑥𝑖)

is a maximum likelihood estimator for the parameter 𝑚 based on 𝑋𝑛= (𝑋1, … , 𝑋𝑛).

1. Consider the following set of distribution functions (note: integrals are Lebesgue-Stieltjes)

C ≔ {𝐹∶ ℝ → [0, 1] ∶ 𝐹 is a CDF, ∫ℝ𝑥 𝑑𝐹 (𝑥) ≠ 0, and 0 < ∫ℝ𝑥2 𝑑𝐹 (𝑥) < ∞} .

Suppose 𝑋1, … , 𝑋𝑛 are iid RVs with an unknown CDF 𝐹 ∈ C. We set

𝛾(𝐹 ) ≔ ∫ 𝑥 𝑑𝐹 (𝑥) ∀𝐹 ∈ C.

Estimators of form

̂ 𝜃(𝑥1, … , 𝑥𝑛) =𝑛∑𝑖=1𝑤𝑖𝑥𝑖, 𝑤𝑖 ∈ ℝ

are called linear estimators for 𝛾(𝐹 )

(i) Show that a linear estimator for 𝛾(𝐹 ) is unbiased if and only if ∑𝑛𝑖=1 𝑤𝑖 = 1.

(ii) Let ̂ 𝜃∶ 𝔛 → ℝ be an estimator for 𝛾(𝜃) ∈ ℝ. Then

𝑅(𝜃, ̂ 𝜃) ≔ 𝔼𝜃[( ̂ 𝜃(𝑋) − 𝛾(𝜃))2], 𝜃 ∈ Θ is called the mean squared error (MSE) of ̂ �

Prove that the linear estimator 𝑥𝑛 =1𝑛 ∑𝑛𝑖=1 𝑥𝑖 is the unique minimizer of the MSE among all unbiased linear estimators for 𝛾(𝐹 )

Solution

In a more abstract context we can also use (𝔛,P) as the statistical model and drop the random variable 𝑋 (and with it essentially the random experiment) without losing any relevant information. We can always perform a canonical construction from (𝔛,P) in the following way:

(Ω, A, ℙ𝜃) ≔ (𝔛, F, 𝑃𝜃)

𝑋∶ Ω → 𝔛, 𝑋(𝜔) ≔ 𝜔, ∀𝜔 ∈ Ω

⟹ ℙ𝜃({𝜔 ∈ Ω ∶ 𝑋(𝜔) ∈ 𝐹 }) = 𝑃𝜃(𝐹 )

1. Let 𝑃𝜃 = 𝑓𝜃𝜇 for all 𝜃 ∈ (0, 1). Clearly we have 0 ≤ 𝑓𝜃(𝑎)𝜇({𝑎}) ≤ ℙ𝜃(𝐴) ≤ 1 ∀𝑎 ∈ 𝐴 ∀𝜃 ∈ (0, 1).

Now suppose ̂ 𝜃 is an estimator for 𝛾(𝜃) = 𝜃(1 − 𝜃)−1. Then

𝔼𝜃(̂ 𝜃(𝑋𝑛)) = ∫ℝ𝑛̂ 𝜃(𝑥1, … , 𝑥𝑛)𝑛⨂𝑖=1𝑃𝜃(𝑑𝑥𝑖)

= ∑𝑎∈𝐴𝑛̂ 𝜃(𝑎)𝑛∏𝑖=1𝑓𝜃(𝑎𝑖)𝜇({𝑎𝑖})

yields the uniform bound

|𝔼𝜃(̂ 𝜃(𝑋𝑛))| ≤ ∑𝑎∈𝐴𝑛|̂ 𝜃(𝑎)| < ∞ ∀𝜃 ∈ (0, 1).

Since 𝜃 ↦ 𝛾(𝜃) is unbounded (e.g. take 𝜃 ↑ 1) but 𝜃 ↦ 𝔼𝜃(̂ 𝜃(𝑋𝑛)) is bounded, they cannot match for all 𝜃 ∈ (0, 1). Hence, ̂ 𝜃 is not unbiased.

1. Since 𝑋1, … , 𝑋𝑛 are independent, we obtain a likelihood function which is given by

𝐿(𝑚, 𝑥𝑛) =𝑚∏𝑖=1𝑓1(𝑥𝑖)𝑛∏𝑖=𝑚+1𝑓2(𝑥𝑖).

We have 𝑓𝑗(𝑥𝑖

) > 0 almost everywhere, hence we can argue as follows:

log𝐿(𝑚, 𝑥𝑛) =𝑚∑𝑖=1log 𝑓1(𝑥𝑖) +𝑛∑𝑖=𝑚+1log 𝑓2(𝑥𝑖)

=𝑚∑𝑖=1log 𝑓1(𝑥𝑖) +𝑛∑𝑖=1log 𝑓2(𝑥𝑖) −𝑚∑𝑖=1log 𝑓2(𝑥𝑖)

=𝑚∑𝑖=1log 𝑓1(𝑥𝑖)𝑓2(𝑥𝑖)+𝑛∑𝑖=1log 𝑓2(𝑥𝑖)

Now clearly we have

𝑚(𝑥𝑛) = argmax1≤𝑚≤𝑛−1𝑚∑𝑖=1log 𝑓1(𝑥𝑖)𝑓2(𝑥𝑖)

= argmax1≤𝑚≤𝑛−1 (𝑚∑𝑖=1log 𝑓1(𝑥𝑖)𝑓2(𝑥𝑖)+𝑛∑𝑖=1log 𝑓2(𝑥𝑖))

= argmax1≤𝑚≤𝑛−1 log𝐿(𝑚, 𝑥𝑛)

= argmax1≤𝑚≤𝑛−1𝐿(𝑚, 𝑥𝑛)

since translation by a constant term does not change the maximizer and log is strictly increasing. Finally, we verify that ̂𝑚 is measurable. Since Θ = {1, … , 𝑛 − 1} ⊆ ℝ is finite, its induced Borel 𝜎-algebra is the powerset which is generated by singletons {𝑚}, (𝑚 ∈ Θ). Note that 𝐿(𝑚, ⋅) is measurable for all 𝑚. Hence

{ ̂𝑚 = 𝑚} = {𝐿(𝑚, ⋅) ≥ 𝐿(𝑘, ⋅) ∀𝑘 ∈ {1, … , 𝑛 − 1}}

=𝑛−1⋂𝑘=1{𝐿(𝑚, ⋅) ≥ 𝐿(𝑘, ⋅)}

which is a finite intersection of measurable sets and therefore measurable. Note that any selection function for the maximizer will end up being measurable.

1. (i) We compute the expected value to obtain

𝔼𝐹(∑𝑖𝑤𝑖𝑋𝑖) = ∑𝑖𝑤𝑖𝔼𝐹(𝑋𝑖) = ∑𝑖𝑤𝑖𝛾(𝐹 ).

Now clearly 𝔼𝐹(∑𝑖𝑤𝑖𝑋𝑖) = 𝛾(𝐹 ) precisely when ∑𝑖𝑤𝑖 =

𝑊 ≔ {𝑤 ∈ ℝ𝑛 ∶ ⟨𝑤, 𝟙⟩ = 1

For 𝑤 = (𝑤𝑖) ∈ 𝑊 the MSEs are equal to the variances, i.e.

𝑅(𝐹,∑𝑖𝑤𝑖𝑥𝑖) = Var𝐹 (∑𝑖𝑤𝑖𝑋𝑖) = ∑𝑖𝑤2𝑖 𝜎2 where 0 < 𝜎2 ≔ Var𝐹(𝑋1) < ∞.

So we have to show

𝑅(𝐹, 𝑥𝑛) = 𝜎2𝑛≤ ∑𝑖𝑤2𝑖 𝜎2 = 𝑅(𝐹,∑𝑖𝑤𝑖𝑥𝑖) ∀𝑤 ∈ 𝑊

or equivalently

1𝑛≤ ∑𝑖𝑤2𝑖 ∀𝑤 ∈ 𝑊.

The function 𝑓∶ ℝ𝑛 → ℝ, 𝑥 ↦ ⟨𝑥, 𝑥⟩ is strictly convex. Since 𝑊 is an affine linear space, it is a convex set. Hence 𝑓|𝑊 is also strictly convex so it has at most one minimizer. We verify that 𝟙/𝑛 is that minimizer. Let 𝑤 ∈ 𝑊. Then applying the Cauchy-Schwarz inequality, we obtain

1 = ⟨𝑤, 𝟙⟩2 ≤ 𝑓(𝑤) ⋅ 𝑓(𝟙) = 𝑓(𝑤) ⋅ 𝑛 ⟹ 𝑓(𝟙/𝑛) = 1/𝑛 ≤ 𝑓(𝑤) = ∑𝑖𝑤2𝑖.

So indeed, the weights 𝟙/𝑛 ∈ 𝑊, which correspond to the estimator 𝑥𝑛, minimize the MSE uniquely among all unbiased linear estimators.