Risk Matrices: Why Two Operators Classify the Same Occurrence Differently

Two safety analysts in a control room discussing a risk matrix classification discrepancy, with an aircraft visible through the window.

Two safety departments receive the same occurrence report. One closes it amber, the other red, and both are right. This is not a failure of one of the two systems — it is what a risk matrix is for. Severity and probability are not properties of the event; they are properties of the barriers that survived it and of the exposure the operator measures. Comparing two matrices without comparing their scales produces a number with no meaning at all.

A matrix is a language, not a measurement

The five-by-five grid that appears in almost every safety management manual reproduces the example scales published in ICAO’s safety management material: a severity scale running from catastrophic to negligible and a likelihood scale running from frequent to extremely improbable. What the same material says, and what gets lost when the grid is copied into a manual, is that the scales are illustrative. The organisation is expected to define its own, calibrated to what it does.

That instruction is not an invitation to be vague. It is a recognition that a helicopter operator flying offshore shuttles, a regional airline and a Part-145 organisation cannot share a probability scale, because they do not share an exposure unit, a fleet size, or a set of credible outcomes. A grid borrowed intact from another operator’s manual is a grid whose anchors describe someone else’s business.

So the first question to ask of any risk classification is not “is this red or amber”. It is “against which scale, defined by whom, anchored to what”.

Severity is what the barriers left, not what the event was called

The most common error in occurrence classification is to read severity off the event’s title. A tail strike is a tail strike; a level bust is a level bust; therefore the severity is fixed. It is not.

Severity in a safety management system is a statement about the credible outcome of the accident scenario that this occurrence belonged to, had it continued. That means the assessor has to name the scenario, not the event. “Level bust” is not a scenario. “Level bust into occupied airspace on a converging track, with the airborne collision avoidance system as the only remaining barrier” is a scenario, and its credible worst outcome is not the same as the identical altitude deviation in empty airspace under radar coverage with a controller watching.

This is why the same words in two reports produce two severities. Both assessors are answering the same question honestly, about two different situations that were given the same name by the reporting form.

The trap. Classifying by event taxonomy. The taxonomy exists so that occurrences can be counted and grouped; it is not a severity key. As soon as a matrix is fed from the event type rather than from the scenario, the whole system stops measuring risk and starts measuring vocabulary.

A risk matrix doesn’t just look at what happened—it looks at what could have happened if things went wrong. Two operators might classify the same event differently because they’re imagining different worst-case scenarios and different safety barriers. The key is to always ask: “What was the real danger here, and what stopped it?”

An A320 on a short-haul flight has a tail strike during landing. Operator A classifies it as “moderate severity” because the runway was dry and long, with plenty of remaining distance. Operator B classifies it as “high severity” because the runway was wet and short, and the crew had already breached their go-around policy. Same event, different scenarios.

The barrier question, asked properly

The most useful discipline available for this is the two-question approach used in the event risk classification methodology that many operators have adopted alongside their matrix. It asks, in order: if the occurrence had continued, what is the most credible accident scenario it would have produced? And how much barrier capacity was still standing between the occurrence and that outcome?

The second question is the one that does the work. Barriers are not counted; they are assessed for effectiveness. Three procedural barriers that all depend on the same crew noticing the same thing are, functionally, one barrier. A barrier that worked by chance — a controller who happened to be looking at the right sector — is not an effective barrier, it is a fortunate one, and classifying it as effective is how an operator convinces itself that a serious event was routine.

The scales used in these schemes are deliberately non-linear, so that an occurrence with a credible catastrophic outcome and no barriers remaining dominates every other cell in the register. That is an intentional design choice, and it is one of the places two operators diverge: an organisation using a linear five-by-five scale will let a large number of minor events outweigh a single near-catastrophic one in any aggregate view. A non-linear scheme will not.

Probability is a rate, and a rate needs a denominator

The likelihood axis is where the divergence becomes arithmetical rather than judgemental. “Occasional” means nothing until the organisation says occasional per what.

The choice of denominator is not neutral. A short-haul operator flying sectors of an hour and a long-haul operator flying sectors of eleven hours can have identical per-sector rates of approach and landing occurrences and report per-hour rates that differ by an order of magnitude. Neither is lying. They are measuring different things, and only one of the two denominators is causally connected to approach and landing events at all.

Same events, different denominators: why the likelihood cell moves
Denominator Suits which hazards Distorts
Per flight hour Cruise, systems, fatigue-related exposure, maintenance intervals Flatters long-haul on approach, landing and takeoff events
Per sector or per departure Takeoff, approach, landing, ground handling, turnaround Penalises short-haul on anything driven by time aloft
Per cycle Structural, landing gear, pressurisation, brake and tyre events Diverges from sectors when training flying is significant
Per movement at an aerodrome Runway incursions, ground collisions, apron damage Depends on whose movement count is used, and whether general aviation is included
Qualitative, no denominator Nothing, once the organisation has data Everything — it makes the axis a matter of the assessor’s memory

There is a second, quieter distortion on this axis: reporting culture. An operator with a mature just culture and a high reporting rate sees more events and will, on any honest scale, place several hazards higher in probability than a comparable operator whose people do not file. The organisation that looks safer on the grid is frequently the one that knows less. Anyone comparing two registers without comparing reporting rates per thousand sectors is comparing willingness to report, not risk.

The choices that legitimately separate two operators

Set out explicitly, the divergences are structural rather than accidental. Each one is a defensible decision that the organisation is required to document — and each moves the cell.

Where two compliant matrices part company
Choice One house The other Effect on the cell
Severity anchor Most credible outcome Reasonable worst case Typically one to two rows
Unit of assessment The individual occurrence The hazard, with occurrences as evidence Changes what probability even means
Barrier accounting Barriers counted Barriers assessed for effectiveness Fortunate outcomes reclassified upward
Exposure denominator Flight hours Sectors or cycles Up to an order of magnitude on the likelihood axis
Scale shape Linear five by five Non-linear weighted score Changes which risks dominate the register
Tolerability boundary Drawn conservatively Drawn where resource allows action Determines what must be escalated
Assessor Reporter, then reviewed Safety review group only Changes the spread, not the average

The same occurrence, run through two houses

Take a single, entirely ordinary report: an unstabilised approach continued below the operator’s gate, followed by a normal landing, filed voluntarily by the crew.

One report, two defensible classifications
Question Operator A Operator B
Scenario named Long landing, runway excursion Loss of control in the flare, or excursion
Severity anchor used Most credible outcome — major Reasonable worst case — catastrophic
Barriers still standing Go-around policy, long dry runway, ample landing distance Go-around policy already breached; runway wet, limiting
Barrier effectiveness Limited but real Minimal — the crew flew through the only gate
Exposure denominator Per 1000 sectors, and this is the fourth this quarter Per flight hour, and this is the first this year
Resulting cell High probability, moderate severity — amber, trend watched Low probability, high severity — red, immediate action

Nothing in that table is an error. Operator A is running a trend analysis on a recurring behaviour; Operator B is running a barrier analysis on a single event. Both will act. They will act differently, and the value of the exercise lies in the actions, not in the colour.

The good reflex. Before arguing about a cell, make the assessor state the accident scenario out loud and name the barriers that were still standing. Most classification disputes dissolve at that point, because the two people were assessing two different scenarios without knowing it.

Ordinal cells, cardinal decisions

There is a methodological objection to risk matrices that safety departments should know, because it explains a class of bad decision that no amount of care in filling the grid will prevent. The cells of a matrix are ordinal: red is worse than amber, row four is worse than row three. They are not cardinal. Row four is not twice row two.

The moment an organisation multiplies its severity rank by its likelihood rank to produce a risk score, then ranks its whole register on that product, it has performed arithmetic on labels. Two events with the same product can differ by orders of magnitude in expected harm, and a matrix can rank a genuinely worse hazard below a milder one. The objection is well established in the risk literature, and it is an objection to the arithmetic, not to the assessors.

The practical defence is modest but effective: use the matrix to decide what needs attention and at what level it is escalated, never to rank-order a portfolio by score, and never to compare a score across two registers built on different anchors.

What has to be harmonised before any comparison

Comparison between operators is not impossible — it is conditional. Regulators discovered this early, which is why occurrence reporting in Europe carries a common taxonomy and a common risk classification scheme layered on top of each operator’s internal matrix. The point of that layer is not to replace the operator’s judgement; it is to give the authority one scale on which occurrences from a hundred different management systems can sit side by side.

Internally, the same logic applies to any merger, wet-lease arrangement or code-share safety review. Before the two registers are placed on the same page, four things have to be reconciled: the severity anchor, the barrier effectiveness definitions, the exposure denominator, and the position of the tolerability boundary. Reconcile those and the comparison is informative. Skip them and the exercise produces a chart that looks authoritative and means nothing — the most expensive kind of safety product there is.

Writing the report so the classification survives

Most of the difficulty described here is created at the point of entry, not at the point of assessment. A report that says what happened but not what nearly happened forces the assessor to invent the scenario, and two assessors will invent two.

A man in a uniform reviews a risk matrix on a computer screen in an airport control room.
Risk assessments require consistent criteria to avoid subjective interpretations.

The fields that make a classification robust are the ones that describe the situation rather than the event: what else was in the airspace or on the runway, which defences were still available, which of them the reporter was aware of at the time, what the reporter believes would have happened had they not acted, and how close the next barrier was. Safety departments comparing practice across operators — the kind of exchange that happens in the safety threads on the FL330 forums — consistently find that the quality of those fields predicts the consistency of the classification far better than the design of the grid does.

A risk matrix does not measure risk. It records a set of judgements about scenarios, barriers and exposure, in a shorthand that lets an organisation decide what to look at next. Read that way, two operators classifying the same event differently is not a discrepancy to be resolved. It is two organisations correctly describing two different sets of defences — and the only real error would be assuming they were describing the same thing.

Commentaires

Leave a Reply

Découvrez l’écosystème FL330

FL330, c’est toute l’aviation professionnelle — explorez nos autres espaces.