Type I and II errors and test decisions: VCE Specialist Mathematics Units 3 & 4

1. A decision can be wrong in two distinct ways

A hypothesis test uses a random sample to choose between rejecting H₀ and not rejecting H₀. The true population state is fixed but generally unknown. Combining those two possible decisions with whether H₀ is actually true gives two correct outcomes and two kinds of error.

A Type I error occurs when H₀ is true but the test rejects it. A Type II error occurs when H₀ is false, at a value in the alternative being investigated, but the test does not reject it. If a true null is not rejected, or a false null is rejected, the decision is correct for that population state.

The error type is not determined by the p-value alone. A small p-value leads to rejection, but whether that rejection is a Type I error depends on the unknown truth. Similarly, a non-rejection might be correct or might be a Type II error. A test reports the evidence and decision; it does not reveal with certainty whether an error has occurred.

For example, suppose a bottling line is tested for underfilling: H₀: μ = 500 mL against H₁: μ < 500 mL. A Type I error means concluding that the line is underfilling when its true mean is 500 mL. A Type II error means failing to detect underfilling when the true mean is below 500 mL. Describe both the decision and the actual population condition.

2. Conditional probabilities make the distinction precise

The significance level controls the Type I error probability under the null model. For an exactly calibrated normal test, the rejection region has null probability α. For a test using a large-sample approximation, the intended significance level is nominal: the actual error probability can differ if the approximation is imperfect.

The Type II probability needs a specific alternative value. A test might rarely miss a large increase but often miss a tiny one. There is therefore no single numerical β determined solely by saying “H₁: μ > 100”. Specify a true alternative mean such as μ = 106 before calculating its probability of non-rejection.

The conditional direction is crucial. Pr(reject H₀ | H₀ true) is not Pr(H₀ true | reject H₀). The latter would require a different inferential framework and additional information. Likewise, a p-value of 0.03 is not a 3% probability that H₀ is true or a 3% probability that the current decision is a mistake.

It is helpful to call 1 − β(μ₁) the probability of detecting that stated alternative, often called power at μ₁. This is simply the complement of missing that same alternative. It is not generally 1 − α, because α and β are calculated under different population assumptions.

3. Fix the rule under H₀, then change the distribution

To calculate a Type II error probability, first derive the critical region using the null distribution and the chosen α. Freeze that region. Next replace μ₀ by the specified true alternative mean μ₁ in the sampling distribution, while keeping any supplied standard deviation assumption. Finally find the probability of landing in the non-rejection region under this new distribution.

Do not redesign the critical boundary around μ₁. The test is still asking whether to reject the original H₀. Moving its boundary to track the alternative changes the decision rule and answers another question. In a diagram, the sampling density shifts while the boundary stays at the same horizontal coordinate.

For an upper-tail test with known σ and sample size n, let SE = σ/√n and c = μ₀ + z_1−αSE. The test rejects for X̄ ≥ c. At a particular true μ₁ > μ₀, the probability of a Type II error is:

For a lower-tail test, the non-rejection event is to the right of its lower boundary. For a two-sided test, the non-rejection region is the interval between two boundaries. Write that event explicitly before standardising so that the correct area follows naturally.

4. Understand the consequences of changing the design

With n, σ and a particular alternative held fixed, lowering α makes rejection harder. For an upper-tail test, the critical boundary moves right. The null rejection area decreases, but the alternative non-rejection area increases: fewer false alarms come with more missed departures. Increasing α reverses this trade-off. Neither error type should be considered without its practical consequences.

Increasing n is a different change. At fixed α and a fixed alternative separated from μ₀, the standard error decreases and the two sampling distributions become easier to distinguish. The Type I probability stays at the chosen α, while the Type II probability decreases in these normal mean tests. More data can therefore improve detection without increasing the planned false-alarm rate.

A larger true departure in the direction of H₁ is easier to detect, all else fixed. Greater population variability makes detection harder because it increases the overlap between possible sampling distributions. These statements describe a specified model; they do not justify treating repeated dependent measurements as if they were a larger independent sample.

5. Plan a sample size for a specified detection target

A non-routine extension asks for the smallest n that keeps α fixed while making β sufficiently small at one chosen alternative. For the upper-tail setup in worked example 1, algebra gives β(106) = Φ[z_0.95 − 0.6√n]. To require β ≤ 0.10, use the increasing nature of Φ:

Therefore n = 24 is the smallest integer meeting this target. Check n = 24 and n = 23 using the original probability formula, recalculating each critical mean at the same α. The guarantee concerns the specified alternative mean 106 under the normal model. It does not imply a 90% detection probability for every mean just above 100.

This investigation combines inverse normal quantiles, inequality reasoning and integer rounding. Keep the distinction between a required β and a required detection probability: detection of at least 90% means β at most 10%. Reversing that complement would produce a design with almost the opposite aim.

6. Simulate each error under the correct truth

To estimate α, generate samples under μ₀ and count rejections. To estimate β(μ₁), generate samples under μ₁ and count non-rejections. The same previously calculated test boundary is used in both experiments. Mixing samples from the two populations without recording which generated them cannot directly estimate either conditional probability.

input n, R, μ₀, μ₁, σ, α
SE ← σ / sqrt(n)
c ← μ₀ + inverseNormal(1 − α) × SE
missed ← 0
for r from 1 to R
    sample ← n independent normal draws with mean μ₁ and SD σ
    mean ← average(sample)
    if mean < c then
        missed ← missed + 1
    end if
end for
output missed / R

A rare error needs many repetitions to estimate accurately. In worked example 2, a simulation with 1000 repetitions will usually show no misses even though the true probability is positive. Zero simulated errors is not proof of impossibility. Use the analytic normal calculation for a precise small probability and simulation to check the overall logic.

In a contextual conclusion, name the error's consequence without making an unsupported policy decision. A false alarm may cause an unnecessary inspection; a missed shift may allow a faulty process to continue. Which is more costly depends on information outside the test calculation. Statistical design makes the competing risks explicit rather than deciding their values automatically.