Trustworthy Numerical Computing

Trustworthy numerical results require more than code that compiles, runs, and passes ordinary tests. Scientific software must also account for finite precision, problem conditioning, algorithmic stability, convergence, validation, and variability across computing environments.

This training develops a practical workflow for recognizing, investigating, preventing, and communicating numerical reliability problems. The concepts are language-agnostic and apply to scientific work in Python, Julia, C, C++, Fortran, R, MATLAB, Rust, and similar environments.

Learning outcomes

When you complete this training you will be able to

  • explain how floating-point representation and rounding affect computations;
  • choose meaningful error measures and numerical tolerances;
  • distinguish an ill-conditioned problem from an unstable algorithm or an implementation defect;
  • propagate declared input bounds or standard uncertainties through a smooth model and recognize when first-order propagation is inadequate;
  • recognize common failure modes such as cancellation, overflow, underflow, and accumulated rounding error;
  • define defensible convergence and stopping criteria;
  • validate computations using references, invariants, properties, refinement, and independent methods;
  • reason about reproducibility across compilers, hardware, optimizations, and parallel execution;
  • report numerical assumptions, limitations, and validation evidence clearly.

Learning path

The learning modules are the self-contained reading material for the course. They follow a progression from the arithmetic model to diagnosis, validation, reproducibility, and communication.

The slide deck supports instructor-led delivery. It provides teaching prompts, short examples, and discussion anchors rather than duplicating the complete reading material.

The self-paced Jupyter activities begin with the opening numerical experiment, in which two plausible variance calculations lead to opposite scientific decisions. The follow-up floating-point landmarks activity explains that discrepancy by examining representable spacing, rounding, special values, and evaluation order. The comparison criteria across scales activity turns observed discrepancies into reviewable decisions using reference quality, units, error measures, and justified tolerances. The sensitivity, stability, and residuals activity then separates intrinsic problem sensitivity from avoidable algorithmic error and implementation defects. The failure-mode laboratory applies that diagnosis to cancellation, summation order, overflow, underflow, scaling, and log-domain reformulations. The convergence and stopping diagnostics activity separates forward error, residual, and update size; classifies convergence and failure; and uses a tolerance study to find an attainable binary64 accuracy floor. The validation evidence portfolio activity combines exact cases, method properties, refinement rates, convexity bounds, and a separately bounded series while keeping code verification, solution verification, and model validation distinct. The environment variation and reproducibility activity then compares reduction order, partition, and accumulator precision; it separates harmless low-order variation from changes that reverse a scientific conclusion. The numerical reliability statement activity converts reference, refinement, input-range, and environment evidence into a concise claim with supported digits, a decision margin, and explicit limitations. The final sensor-inversion capstone then asks participants to reproduce a precision-dependent threshold decision, diagnose an ill-conditioned component split, propagate deterministic sensor bounds, improve the calculation and decision contract, and write a qualified reliability statement.

Suggested schedule

The modules are designed so that the core material can be delivered as a one-day course or split over two half-days. Allow about 75 minutes for the complete capstone investigation; selected stages can be assigned as follow-up work in a shorter delivery.

Course block Modules
Motivation and arithmetic foundations 1–3
Diagnosing and controlling numerical error 4–6
Validation and reproducibility 7–8
Communication and capstone investigation 9–10

Target audience

This training is for scientists, research software engineers, and technical programmers who develop, review, or rely on numerical software and want stronger evidence that a computed result is scientifically defensible.

Prerequisites

Participants should already be able to read and modify small programs in at least one scientific-computing language. This course does not teach programming from scratch.

You should be comfortable with:

  • variables, expressions, functions, loops, and arrays;
  • basic algebra and scientific notation;
  • running a short program or notebook and inspecting its output;
  • interpreting tables and simple plots;
  • the idea that measurements and models have limited precision.

Prior knowledge of IEEE 754, numerical analysis, parallel programming, or a specific implementation language is not required.

Quick self-assessment

You are likely ready if you can:

  • implement or recognize a loop that sums a collection of numbers;
  • compare the output of two implementations;
  • explain why a calculated result should not report more meaningful digits than its inputs support;
  • run the same calculation with a changed input or resolution;
  • describe what evidence would make you trust a scientific result.

Software and access requirements

The reading material and slides only require a web browser. Hands-on work uses short Python programs and Jupyter notebooks. Each published activity page provides both a notebook download and an Open in Google Colab button for a hosted Python environment. Local work uses the course environment described in SETUP.md. Participants do not need Quarto; it is used by developers and maintainers to build and preview the published material. The numerical concepts remain applicable across scientific programming languages.

Level of the material

For participants with the prerequisite programming background, the planned material is approximately:

  • Introductory: 25%
  • Intermediate: 55%
  • Advanced: 20%

Trainer

Geert Jan Bex