A program can run correctly and still produce a result that should not be trusted. In this experiment, two mathematically equivalent calculations give different answers to the same scientific question.
This activity accompanies Module 1: When Correct Code Produces Wrong Answers.
Download the executable Jupyter notebook Open in Google Colab
After this experiment, you should be able to:
A sensor records four event times in nanoseconds. We want the population variance of those times as a measure of their spread. The monitoring rule flags the run when the variance is greater than \(22.25\ \mathrm{ns}^2\).
For this numerical experiment, treat the four recorded values as exact. Measurement uncertainty and whether variance is the best scientific metric are important, but deliberately outside the scope of this first investigation.
Measurements (ns): [100000004.0, 100000007.0, 100000013.0, 100000016.0]
Flag the run above: 22.25 ns²Two colleagues implement algebraically equivalent formulas. For \(n\) values with mean \(\bar{x}\):
Before running the next cell, predict:
def variance_centered(values):
"""Population variance using deviations from the computed mean."""
mean = sum(values) / len(values)
return sum((value - mean) ** 2 for value in values) / len(values)
def variance_shortcut(values):
"""Population variance using an algebraically equivalent identity."""
mean = sum(values) / len(values)
mean_of_squares = sum(value**2 for value in values) / len(values)
return mean_of_squares - mean**2
candidate_results = {
"centered formula": variance_centered(measurements_ns),
"shortcut formula": variance_shortcut(measurements_ns),
}
for method, value in candidate_results.items():
decision = "FLAG" if value > variance_limit_ns2 else "accept"
print(f"{method:18s}: {value:5.1f} ns² -> {decision}")centered formula : 22.5 ns² -> FLAG
shortcut formula : 22.0 ns² -> acceptBoth functions terminate normally and return ordinary numbers. Both values look plausible for measurements whose deviations differ by only a few nanoseconds, yet they lead to opposite monitoring decisions. An error of \(0.25\ \mathrm{ns}^2\) is already large enough to change the classification at this threshold, so the required numerical accuracy is scientifically consequential.
Do not choose a result because its implementation looks more sophisticated or because its output has more digits. We need evidence tied to the numerical question.
Python’s Fraction type performs exact rational arithmetic. Because the inputs in this exercise are exact integer-valued nanoseconds, it can provide a reference that does not incur floating-point rounding during the calculation.
This is useful evidence for this small case; it is not a general claim that exact arithmetic is practical for every scientific computation.
from fractions import Fraction
def exact_population_variance(values):
exact_values = [Fraction(value) for value in values]
exact_mean = sum(exact_values, start=Fraction(0)) / len(exact_values)
squared_deviations = (
(value - exact_mean) ** 2 for value in exact_values
)
return sum(squared_deviations, start=Fraction(0)) / len(exact_values)
exact_variance = exact_population_variance(measurements_ns)
reference_variance_ns2 = float(exact_variance)
print(
f"Exact variance: {exact_variance} ns² = "
f"{reference_variance_ns2} ns²"
)
for method, value in candidate_results.items():
absolute_error = abs(value - reference_variance_ns2)
print(f"{method:18s}: absolute error = {absolute_error:.1f} ns²")Exact variance: 45/2 ns² = 22.5 ns²
centered formula : absolute error = 0.0 ns²
shortcut formula : absolute error = 0.5 ns²Adding the same constant to every measurement changes their location but not their spread. The population variance should therefore be shift invariant.
The next cell keeps the event times relative to a common offset [4, 7, 13, 16] fixed and changes only that common offset. Predict how each implementation should behave before running it.
relative_times_ns = [4.0, 7.0, 13.0, 16.0]
offsets_ns = [
0.0,
10_000.0,
1_000_000.0,
100_000_000.0,
1_000_000_000.0,
10_000_000_000.0,
1_000_000_000_000.0,
]
print(
f"{'offset (ns)':>18s} "
f"{'centered (ns²)':>15s} "
f"{'shortcut (ns²)':>15s}"
)
for offset_ns in offsets_ns:
shifted_values = [
offset_ns + relative_time for relative_time in relative_times_ns
]
centered = variance_centered(shifted_values)
shortcut = variance_shortcut(shifted_values)
print(f"{offset_ns:18.0f} {centered:15.1f} {shortcut:15.1f}") offset (ns) centered (ns²) shortcut (ns²)
0 22.5 22.5
10000 22.5 22.5
1000000 22.5 22.5
100000000 22.5 22.0
1000000000 22.5 0.0
10000000000 22.5 16384.0
1000000000000 22.5 0.0The centered implementation agrees with the exact reference and preserves the expected variance across these offsets. The shortcut implementation changes as the common offset grows; it eventually reports values that are zero or much larger than the complete spread in the data.
That evidence is enough to reject the shortcut implementation for this decision. It does not yet explain the arithmetic mechanism. Module 2 develops the finite-precision model needed for that explanation.
Change test_offset_ns and rerun the cell. Find the smallest power of ten for which the shortcut result no longer agrees with the exact reference to within \(0.1\ \mathrm{ns}^2\).
test_offset_ns = 10_000_000.0 # Try 10**k for different integer values of k.
test_values = [
test_offset_ns + relative_time for relative_time in relative_times_ns
]
test_results = {
"centered": variance_centered(test_values),
"shortcut": variance_shortcut(test_values),
"exact reference": float(exact_population_variance(test_values)),
}
test_results{'centered': 22.5, 'shortcut': 22.5, 'exact reference': 22.5}A compact record makes the conclusion reviewable. Notice that it includes the question, decision threshold, reference, observed discrepancy, invariant, and limitation—not just the preferred number.
evidence = {
"question": "Is the population variance above 22.25 ns²?",
"assumption": "Input event times are exact for this experiment.",
"exact_reference_ns2": reference_variance_ns2,
"centered_result_ns2": candidate_results["centered formula"],
"shortcut_result_ns2": candidate_results["shortcut formula"],
"independent_check": "Variance should be unchanged by a common offset.",
"limitation": (
"Measurement uncertainty and metric choice were not assessed."
),
}
evidence{'question': 'Is the population variance above 22.25 ns²?',
'assumption': 'Input event times are exact for this experiment.',
'exact_reference_ns2': 22.5,
'centered_result_ns2': 22.5,
'shortcut_result_ns2': 22.0,
'independent_check': 'Variance should be unchanged by a common offset.',
'limitation': 'Measurement uncertainty and metric choice were not assessed.'}For the four exact event times, the centered calculation gives a population variance of \(22.5\ \mathrm{ns}^2\). It agrees with an exact rational reference and remains unchanged under the tested common offsets. The shortcut calculation gives \(22.0\ \mathrm{ns}^2\) for the original data and violates shift invariance as the offset grows, so it is not reliable for the \(22.25\ \mathrm{ns}^2\) decision threshold. This experiment does not assess measurement uncertainty or whether variance is the best scientific metric.
Compare this statement with “the answer is 22.5.” Which parts make the conclusion easier to review or challenge?
statistics.pvariance from Python’s standard library and decide whether agreement alone makes it an independent reference.