Safety performance indicators: escaping the decorative dashboard

Salle d'analyse de la sécurité d'un exploitant, écrans muraux affichant des courbes et des cartes de chaleur

Many safety dashboards mostly measure their author’s ability to fill in a table. They display rates that never trigger anything, compare quarters without allowing for statistical noise, and let the safety committee conclude that ‘the trend is good’. A useful indicator has one simple property: somebody knows what they will do when it crosses its threshold.

What the framework actually requires

A safety management system rests on four components: policy and objectives, risk management, safety assurance, promotion. Measurement belongs to assurance: it exists to verify that the controls really work, not to produce a report. The European framework writes it into the management system requirements applicable to operators, and ties it to the occurrence reporting regulation, which mandates collection, analysis and follow-up — and protects the person who reports.

The consequence is often forgotten: a safety performance indicator only means something when it is attached to an objective and an alert level defined before the measurement. Choosing the threshold after seeing the data is describing the past.

Lagging indicators, leading indicators

Two families, two uses
Lagging (outcome) Leading (precursor)
Measures What happened What makes it more likely to happen
Examples Runway excursions, late go-arounds, injuries Unstabilised approaches, configuration deviations, delayed task completion
Frequency Low — statistically poor High — usable
Risk in use Drawing conclusions from noise Measuring what is easy to count
Role Verifying that objectives are met Steering day to day

A dashboard made only of lagging indicators steers nothing: the events are too rare to tell an improvement from a fluctuation. A dashboard made only of leading indicators drifts towards whatever is convenient to count. You need both, and you need to state the causal link assumed between them — that link is what justifies the indicator, not the availability of the data.

Building an indicator that holds

Six elements, without which an indicator is not an indicator but a number.

Two workers in high-visibility vests monitor safety data on screens at a control station with an airport tarmac visible outside.
Real-time data drives proactive safety management.
  1. The operational definition. ‘Unstabilised approach’ must translate into criteria measurable on flight data: parameters, altitude gates, tolerances. Two analysts must arrive at the same count.
  2. The denominator. A number of events means nothing without exposure: per thousand sectors, per thousand hours, per thousand movements. A change of denominator explains half the ‘trends’ observed.
  3. The window. A twelve-month rolling rate damps seasonality; a monthly rate reacts fast but swings. Both can be read, never one for the other.
  4. The reference. A baseline period, long enough to estimate the mean and the spread.
  5. The alert level. A common practice is to set it at one, two or three standard deviations above the reference mean, with a triggering rule — for instance one point beyond three standard deviations, or two consecutive points beyond two.
  6. The action attached to it. Who does what when the threshold is crossed. Without that line, the indicator is decorative.

The test that eliminates 80% of useless indicators. Ask the question: ‘if this indicator doubles next month, who decides what?’ If the answer is ‘we’ll discuss it at the committee’, the indicator is not ready. If there is no answer, take it off the dashboard — a shorter, operational dashboard beats an exhaustive, inert one.

The costliest statistical traps

Concluding from small numbers

Going from 2 to 4 events in a quarter is not a doubling of risk: it is a variation consistent with chance for a rare phenomenon. Treating that variation as a signal consumes resources and, worse, discredits the system when the figure comes back on its own.

The denominator that moves

A fleet picking activity back up sees its rates fall mechanically at a constant number of events. Exposure must therefore be tracked too, and any improvement coinciding with a rise in activity treated with suspicion.

The numerical target that distorts the measurement

A target to reduce the number of reports is the perfect counter-example: it is met by reporting less. Any target set on an indicator fed by voluntary reporting creates pressure to under-report. Targets must bear on rates of undesirable events measured objectively, never on reporting volume.

Under-reporting is the main failure mode of a safety management system. A fall in reports reads as an improvement on a dashboard; it often means that just culture has degraded. Two safeguards: track the reporting rate as an indicator of system health and not of safety, and measure the turnaround time back to the reporter, which predicts the willingness to report again better than any speech.

A defensible starting set of indicators

Example framework for an air transport operator
Indicator Family Denominator Action at the threshold
Unstabilised approaches Leading 1,000 approaches Analysis by aerodrome and by procedure; targeted feedback in training
Late go-arounds Lagging 1 000 approches Review of individual events with the crew, under just culture
Take-off configuration deviations Amont 1,000 departures Review of verification procedures and task interruptions
Active maintenance deferrals beyond their interval Amont Per aircraft Planning and parts trade-offs at management level
Turnaround time back to the reporter System health Median, in days Reinforcement of the analysis team; process review
High-potential events Aval 10,000 sectors Management review, re-examination of the controls concerned

Flight data monitoring feeds several of these lines; it is in fact required above a certain mass threshold for transport aeroplanes, and its use must remain non-punitive to produce anything. That is a technical condition as much as an ethical one: a flight data monitoring programme used for disciplinary purposes quickly stops being informative, because behaviour adapts to the measurement.

Tie every indicator to a control

This is the step that separates a dashboard from a steering instrument. A risk analysis describes, for a feared event, the barriers that prevent it and those that limit its consequences. Every important barrier should have an indicator attesting that it still works — and that indicator measures the state of the barrier, not the occurrence of the event.

Take the landing runway excursion. The usual barriers are approach stabilisation, the go-around decision, the landing distance calculation in the conditions of the moment, the quality of the runway state information, and braking technique. A dashboard that tracks only the number of excursions measures none of those five barriers: it waits for all of them to give way. A useful dashboard tracks the unstabilised approach rate, the proportion of go-arounds decided at the right moment, the fraction of flights where the calculation was redone on the freshest information.

The attachment rule. For each indicator, write in one line the barrier it accounts for. An indicator that attaches to no barrier measures something nobody has established as mattering; a barrier with no indicator is an assumption nothing verifies. Both show up at a glance on a barriers × indicators matrix — and that matrix is the real deliverable of safety assurance.

What the management review must see, and nothing more

A management review has no business going through thirty curves. It needs four things, and the system must be able to produce them without extra work:

The useful minimum for a management review
What is presented What management does with it
The period’s exposure: sectors, hours, movements Read every rate without mistaking the denominator
Alert level crossings, with the rule applied Note what triggered, and what should have
The state of open actions: which, since when, by whom Arbitrate resources; an unclosed action is an untreated risk
Barriers with no indicator, and indicators with no action Decide to create one, or to remove the other

The fourth line is the one that gets left out, and it is the one that moves the system forward from one review to the next. Without it the dashboard silts up: things are added, never removed, and reading it becomes a ritual.

Keeping the system alive

A set of indicators goes stale. Three habits keep it alive: review each indicator once a year, asking whether it has triggered anything; drop without regret those that have triggered nothing in two years; and document, for each major control, the indicator attesting that it still works. That last habit is the one that ties measurement to the risk analysis, instead of making them two parallel exercises.

The applicable regulation and your organisation’s management manual take precedence over this article, which sets out principles and not obligations specific to any one operator.

A good safety indicator is like a smoke alarm: it doesn’t just show a number, it triggers a clear action when something is wrong. If your indicator doesn’t make someone act, it’s just a decoration. Real safety indicators measure things that can actually prevent accidents, not just count them after they happen.

An airline tracks “unstabilised approaches” (leading indicator) and “runway excursions” (lagging indicator). When the unstabilised approach rate rises above 3 per 1,000 landings, the training department reviews approach briefings. If excursions happen, they investigate runway conditions and crew decision-making. The leading indicator helps prevent excursions before they occur, while the lagging one confirms if the system is working.

Questions fréquentes

How many safety performance indicators do you need?

Few, and every one armed with an action. A framework of six to ten indicators, each with a threshold and a decision-maker, beats thirty lines tracked without consequence. The right criterion is not coverage, it is the ability to trigger a decision.

Where do you set an alert level when you have no history?

You start by building a reference period — often twelve months — publishing the indicator without a threshold, then set the alert from the observed mean and spread. Setting an arbitrary threshold before knowing the spread produces either permanent alerts or none at all.

Is a fall in the number of reports good news?

Almost never. Reporting volume measures the health of the reporting system, not the level of safety. A fall must trigger an examination of just culture, of the turnaround time back to reporters and of the ease of reporting, before any favourable interpretation.

Can you set a numerical target for reducing events?

On events measured objectively, yes. On indicators fed by voluntary reporting it is counter-productive: the target is met by reporting less. The rule is never to target what you ask people to report freely.

Can flight data monitoring be used to identify a crew?

Its use must remain non-punitive and de-identification is the rule, with framed and agreed access arrangements. Beyond the requirement, it is a condition of effectiveness: a programme perceived as disciplinary loses its informative value because practices adapt to the measurement.

Commentaires

Leave a Reply

Découvrez l’écosystème FL330

FL330, c’est toute l’aviation professionnelle — explorez nos autres espaces.