Verification and validation

Two words that get used as if they were one, and the difference between them is the difference between “the software works” and “the answer is right”. They are separate questions, they are answered by different activities, and neither one substitutes for the other.

This page is the vocabulary the rest of the manual uses. The Validation section is written in these terms, the verification checklist in the report is named for one of them, and the distinction is what keeps a conversation about accuracy honest when somebody asks how much to trust a number.

The two questions

Verification asks: are the equations being solved correctly, and is this case set up the way it was meant to be? It is a question about the software and about the case. It is answered internally — no measurement of the real world is involved, and no external data is needed.

Validation asks: does the answer match physical reality? It is a question about the model. It cannot be answered internally at all. It requires data somebody else produced by measuring something.

flowchart TD
  R["A number on the screen"]
  R --> V["VERIFICATION<br/>Are the equations being<br/>solved correctly?"]
  R --> W["VALIDATION<br/>Does the answer match<br/>physical reality?"]

  V --> V1["Does the case generate,<br/>mesh and run?"]
  V --> V2["Is the answer independent<br/>of the mesh?"]
  V --> V3["Have the balances closed<br/>and the monitors settled?"]

  W --> W1["Compared against published data<br/>somebody else produced"]
  W --> W2["Compared against a measurement<br/>of your own machine"]

  V1 --> A["Answered continuously,<br/>inside the software"]
  V2 --> A
  V3 --> A
  W1 --> B["Answered only by a benchmark,<br/>written up so it can be repeated"]
  W2 --> B

The classical shorthand — from the aerospace and ASME literature where this distinction was codified — is verification is solving the equations right; validation is solving the right equations. It is a good summary as long as you remember that “the equations” includes everything you chose: the turbulence model, the boundary conditions, the assumption that density is constant.

Verification: is it being solved right

Verification has two halves, and only one of them is anybody else’s job.

The half the software does

Every solver in the catalogue is exercised on every pass: its case is generated, meshed and run. Not inspected, not reasoned about — run, with the log kept. A solver that stops at start-up because the application failed to write a dictionary it needs is a verification failure, and it is found by trying rather than by assuming.

This is deliberately unglamorous work and it is where the faults actually are. The first time the product was made to reproduce a published answer it turned up two defects that had been present the whole time: every case using the PISO algorithm stopped at start-up, and every closed domain — a cavity, a tank, a sealed vessel — stopped because the pressure level was never pinned. Both had been in the software indefinitely, because nothing had previously demanded that a specific case produce a specific number.

What this half of verification proves is narrow and worth stating exactly: the software can author and execute the case. It says nothing about whether the answer is any good.

Where each solver currently stands is recorded in docs/solver-map.md, including which failures are faults in the product and which are artefacts of the test harness. The distinction is kept because publishing a raw failure count that includes harness noise would be dishonest in the opposite direction.

The half only you can do

The rest of verification is about your case, and the software can only hand you the evidence.

Check	What it establishes	Where
Mesh independence	The answer is a property of the flow, not of the mesh	Mesh independence
Wall resolution	The turbulence model is being used in the regime it is valid in	y+ and wall treatment
Iterative convergence	The discrete equations are actually satisfied	Convergence and residuals
Balances	What goes in comes out; the domain does not leak	Report → verification checklist
Steadiness	A steady answer was appropriate at all	Steady or transient

The report’s “Can these numbers be trusted?” table runs the ones it can compute automatically and — importantly — states the number and the threshold behind every verdict rather than printing a word. A checklist that says “converged” without saying against what cannot be argued with, and something you cannot argue with is not evidence.

Validation: does it match reality

Validation is comparison against data the software had no hand in producing. That is the whole of it, and the constraint is what gives it force.

A benchmark counts as validation only if:

The reference answer came from somewhere else. A published experiment, or a published high-fidelity computation that has itself been checked against experiment. Agreeing with a previous run of the same software is not validation; it is a regression test, which is useful and is not the same thing.
The comparison is quantitative, station by station. “Close” is not a result. A table of values, a stated difference, and a statement of what the remaining difference is attributable to.
The set-up is stated in full, so a reader can build the same case and check.
It says what it does not cover. A cavity benchmark checks incompressible laminar flow, the pressure–velocity coupling, wall conditions and 2-D handling. It says nothing whatever about turbulence models, free surfaces or combustion, and a validation page that leaves that unsaid invites the reader to assume otherwise.

The record of what has been validated, against which published reference, and to what agreement, is manual/05-validation/. That section is the answer to “how do you know it is right”, and it is deliberately shorter than the list of things the product can do.

A benchmark only counts if it is written up

This is the part that gets skipped, so it is worth being blunt about.

More gets run during development than gets documented. An undocumented run is not a result. It cannot be cited in a proposal, defended in a design review, or handed to somebody who asks — because there is nothing to hand them. The knowledge that “we ran that once and it looked right” lives in one person’s head and evaporates.

So a benchmark is not finished when the solver reaches End. It is finished when somebody can open the document, read the set-up, reproduce the case, and get the same table. Until then it does not exist, and the honest thing to say is that it does not exist.

Why “it ran and it looked plausible” is neither

This is the most common failure mode in CFD and it is worth naming, because it feels like success at the time.

A run completes. The residuals fell. The contour plot has a wake behind the body and higher pressure at the nose, which is where they belong. Everything about it looks like a result.

It is not verification, because:

Falling residuals only say the iterative process converged on something. They say nothing about whether the mesh was fine enough, and a mesh-dependent answer converges just as tidily as a mesh-independent one. See Convergence and residuals.
A steady solver on an unsteady flow will plateau and look settled forever. See Steady or transient.
A case with a boundary condition on the wrong patch converges beautifully to the answer to a different question.

It is not validation, because plausible is a judgement about whether the picture matches your expectations, and your expectations are the thing being tested. Colourful and directionally sensible is the floor, not the ceiling. A plot that looks wrong is decisive — it tells you to stop. A plot that looks right tells you nothing.

The reason this matters commercially rather than only academically: the number you take from a plausible-looking run will be quoted in a document, sized a component, or set a load case. It will be treated as an engineering value by people who were not in the room when you decided it looked about right.

Your half of the work

No CFD result is evidence until it has been checked against something you independently trust. The software can verify itself; it cannot validate your case, because it has never seen your machine.

Three things count, roughly in order of strength:

A measurement. A tow-tank run, a wind-tunnel number, a pressure tapping, a flow meter, a thermocouple. Even one point. Even a rough one. A single measured number that the simulation reproduces to within its uncertainty does more for confidence than any amount of mesh refinement.

A hand calculation. Not of the whole flow — of something the flow must obey. Total mass through the domain. A momentum balance across a duct. A pressure drop from a friction factor. Drag from a published coefficient for a similar shape and Reynolds number. These catch order-of-magnitude errors and unit errors, which are the errors that actually happen.

A mesh study. The weakest of the three, because it only ever tells you the answer has stopped depending on the mesh — not that it is right. But it is the one you can always do, it is the first thing a reviewer will ask for, and without it the other two are ambiguous: if a result disagrees with a measurement and you have not done a mesh study, you cannot say whether the model is wrong or the mesh is coarse.

A result with none of these is a hypothesis. It may be a good one. It is still a hypothesis, and describing it as anything else is where CFD’s reputation for producing confident nonsense comes from.

Where you act on this

Mesh independence — the study, and how the Comparison node supports it
manual/05-validation/ — the benchmark record, and the format a new one should follow
Reference → Report — what the verification checklist contains and what each threshold is
How-to → Produce a comparison report