Safety Culture Surveys: Reading the Results Without the Easy Reassurance

A group of maintenance workers in a dimly lit hangar hesitate to fill out a safety survey while a supervisor watches, creating a tense atmosphere.

A safety culture survey almost always comes back reassuring. Averages land in the upper part of the scale, the response rate is presentable, and the deck that reaches the board says the organisation is in good shape. That reading is usually wrong — not because the data is bad, but because the mean of a safety culture survey is the least informative number it contains. The information is in the gaps, the dispersion and the silences.

What a survey measures, and what it cannot

A safety culture questionnaire measures perceptions, held by people who chose to answer, at one moment, about things they believe to be true. It does not measure safety, it does not measure risk, and it does not measure whether the management system works. It measures what employees think would happen if they reported something, what they think their managers actually reward, and whether they think anything changes afterwards.

That is a narrow instrument, and its narrowness is precisely its value. Under the safety management framework set out in ICAO Annex 19 and implemented through the European management system requirements that apply to operators, to Part-145 maintenance organisations and to continuing airworthiness organisations, a large part of the system depends on people volunteering information that nobody else has. Hazard reports, occurrence reports, confidential reports and the informal remark to a duty manager are all voluntary acts. A survey is one of the very few ways to test the willingness that the whole edifice assumes.

What it can never do is settle whether that willingness is justified. Employees may believe reporting is safe when it is not, or believe it is dangerous when the policy is sound and merely invisible. Perceptions are the measurement; reality has to be established elsewhere, by cross-checking against what the system records.

A safety survey tells you what people think, not what’s actually happening. If employees feel reporting is risky, the system might be broken—even if the policy says it’s safe. You need to check real reports to know for sure.

At a European airline, cabin crew scored 75% on “I feel safe reporting errors,” while maintenance staff scored 42%. The gap revealed that maintenance teams feared retaliation, despite the company’s just-culture policy. Investigating showed managers often ignored anonymous reports, proving the perception was correct.

The response rate is a signal about the organisation, not a quality score

The response rate is treated as a proxy for confidence, and the temptation is to be pleased when it is high. It is more useful to read it as an observation about how the survey was collected.

A high rate obtained through voluntary, anonymous, off-duty completion is a genuinely good sign: people believed it was worth their time. A high rate obtained by having supervisors chase individuals, by completing forms during a briefing under observation, or by making completion a condition of something else is a different animal entirely. It tells you the organisation is capable of compliance, which nobody doubted, and it contaminates the answers in the direction of comfort.

The reverse also holds. A low rate is not merely a sampling problem. It is a result. If two-thirds of the line pilots did not answer a survey about whether it is safe to speak up, the sensible working hypothesis is not indifference. Non-response is the most under-read datum in the whole exercise, and it has a specific bias: the people who most doubt the confidentiality of the process are, by construction, the least likely to appear in the data.

The trap. Reporting a response rate without stating how the survey was distributed. The same percentage means opposite things depending on whether a manager was standing nearby, and the distribution method is almost never in the summary deck.

Read the gaps before you read the average

One of the most consistent findings in safety climate research is that different populations inside the same organisation give systematically different answers, and that the difference is larger and more stable than any organisation-wide average. Management typically rates the safety climate higher than the front line. Flight crew and cabin crew often diverge on the same question about the same flights. Line maintenance and base maintenance answer differently about time pressure.

An older man in a suit holds a tablet displaying a survey, while a younger man in a safety vest stands with arms crossed, looking skeptical.
Survey results demand critical review, not blind acceptance.

Those gaps are the diagnostic. An organisation whose average is high but whose management-to-front-line gap is wide is not in good shape: it is an organisation where the people accountable for the system do not know what it feels like to work inside it. That, in Westrum’s classic description of information flow, is the difference between an organisation where bad news travels and one where it is filtered on the way up.

Reading the gap between populations
Comparison A wide gap suggests What to look at next
Senior management versus front line Information is being filtered upward; leadership is reading a sanitised picture How reports are summarised before they reach the accountable manager
Flight crew versus cabin crew Asymmetric willingness to raise a concern across the flight deck door Joint training, briefing practice, and how cabin reports are followed up
One base or line station versus the rest A local management effect rather than a company-wide culture Local rostering, staffing, and who has changed there recently
New joiners versus long-serving staff Either a healthy induction or an unlearned resignation, depending on direction Which way the gap points, and how long it takes new staff to converge
Contract or agency staff versus permanent Reporting is perceived as a risk to future work Whether the just-culture policy explicitly covers non-permanent staff

Stratifying this way collides with confidentiality, and the collision is real. Reporting results for a base with a handful of cabin crew, or for a single maintenance shift, can identify individuals as effectively as printing their names. The workable rule is to fix a minimum cell size before the survey opens, publish it, and suppress anything below it — including when the suppressed cell is the interesting one. Deciding after the fact looks exactly like hiding a bad result, whether or not it is.

Dispersion, « don’t know », and the free text

Three parts of the data set are routinely discarded and are worth more than the headline scores.

  • Dispersion. Two questions can both average in the upper part of the scale: one because almost everyone agrees, the other because half the population strongly agrees and a quarter strongly disagrees. The second is a fractured workforce and the first is a settled one, and the average hides the difference completely. Any item with a bimodal distribution deserves attention regardless of its mean.
  • The « don’t know » rate. On items about policy — whether a just-culture policy exists, what happens after a report is filed, who decides on disciplinary action — a high « don’t know » is not neutral. It measures the visibility of a mechanism the organisation believes it has communicated. A policy that exists and is unknown protects nobody, because the protection only works if the person deciding whether to report believes in it at the moment of deciding.
  • The free text. Comment fields are where the specific, actionable material lives, and they are the first casualty of a rushed analysis. They should be coded systematically rather than quoted selectively: count the themes, note which populations raise which, and resist the pull of the single vivid comment. A striking sentence from one respondent will dominate a management discussion out of all proportion to its weight, and it will crowd out a theme raised quietly by forty people.

The right reflex. Before the results are presented, agree in writing what would count as a bad result. Deciding the threshold after seeing the numbers guarantees that the numbers will be interpreted as acceptable.

The items that carry most of the signal

Not all questions are equally diagnostic. A handful of items map directly onto the mechanisms the management system depends on, and those deserve to be read individually rather than folded into a composite score.

Question families and how they are typically misread
Question family What it actually measures Typical misreading
« I would report my own error » Expected personal consequence of self-reporting Read as honesty; it is a forecast about the organisation’s reaction
« Reports lead to visible change » Whether the feedback loop is closed and seen to close Read as satisfaction; it predicts the future report rate
« I know what happens to a report after I file it » Visibility of the process, not trust in it Merged with trust items, which hides a fixable communication problem
« I can raise a concern with a senior colleague » Steepness of the authority gradient in practice Read as a training result; it is largely a local management result
« Schedules and targets do not push me to cut corners » Perceived balance between production and protection Treated as a morale item; it belongs to the risk picture
« Safety is taken seriously by senior management » Reputation of the leadership, weakly linked to behaviour Given the most weight, though it is the least actionable item on the form

The last line is the one to watch. Generic leadership-commitment items score well nearly everywhere and are hard to connect to any measurable outcome. They are comfortable to present and almost impossible to act on, which is exactly why they end up on the first slide.

Cross-check the perception against what the system already records

A survey read on its own is an opinion poll. Read against the organisation’s own operational record, it becomes evidence. The cross-checks are cheap, because the data already exists in the safety management system.

Survey signal against a hard indicator
Survey signal Indicator to compare What a contradiction means
High stated willingness to report Report rate per population, and the proportion that are self-reports Stated willingness is aspirational; the barrier is practical, not cultural
« Reports lead to change » Time to close an action, and how closure is communicated Actions are being closed administratively without anyone noticing
Low perceived production pressure Deferred defect trends, rostering disruption, late changes to the plan Pressure is normalised and no longer perceived as pressure
Confidence in the just-culture policy Number and outcome of cases where the policy was actually applied Confidence rests on a policy that has never been tested in public
Strong scores at one base Local report quality, not report volume Volume can be encouraged; quality reflects whether reporting is meaningful

Report volume deserves a warning of its own. A rising report count is usually presented as a success and it often is — but it can equally mean a new hazard, a new reporting requirement, or a campaign that encouraged low-value entries. Volume without quality assessment is a management metric, not a safety one.

Benchmarking, trend lines and the temptation of the target

Two habits do more damage than any single misreading.

The first is external benchmarking. Comparing a score against an industry figure assumes an identical instrument, an identical population mix and an identical distribution method. Those conditions are rarely met, and the comparison mostly measures methodological differences. Where cross-operator comparison genuinely helps is in the wording of questions and in what other organisations chose to act on — a discussion that happens more usefully between practitioners, including on the FL330 professional forums, than through a purchased benchmark.

The second is turning a score into a target. The moment a safety culture score appears in a management objective, the organisation acquires an interest in the number rather than in the thing it represents. The mechanisms are mundane and effective: reminders timed just after a good month, framing in the introduction, encouragement to the enthusiastic and silence towards the sceptical. The indicator drifts upward and the underlying reality does not move at all.

What genuinely requires a decision at director level

Most survey findings belong to the safety department. A few do not, because they concern things only an accountable manager can change. It is worth separating them explicitly in the presentation, because a report that mixes both usually results in the safety office being asked to fix problems it has no authority over.

  • Anything about the consequences of reporting. The boundary between an acceptable error and unacceptable behaviour is a policy question and a disciplinary question. It cannot be resolved by promotion campaigns, and it needs to be stated by the person who signs disciplinary decisions.
  • Anything about production pressure. Turnaround times, roster stability, staffing margins, tolerance for deferred defects. These are commercial decisions with a safety consequence, and they sit above the safety manager.
  • Anything about the visibility of feedback. Whether outcomes are published, in what form, and with what delay. This is cheap to fix and reliably neglected.
  • Persistent gaps by base or by function. A local pattern that survives two survey cycles is a management issue at that location, not a cultural characteristic of the workforce there.

Re-measuring without teaching people the answer

Repeating a survey is where most of the analytical value accumulates, and where it is most easily destroyed. Keep the wording of the core items stable, or the trend measures the rewrite rather than the organisation. Keep the distribution method stable, for the same reason. Change one thing at a time, and record what changed.

Above all, publish something back before asking again. The single most reliable way to depress the next response rate is to run a survey, say nothing about the results, and change nothing visible. Employees draw the obvious conclusion, and they draw it accurately: this exercise is a measurement of them rather than a mechanism for them. The next set of numbers will look calmer, the response rate will be lower, and the organisation will file it as an improvement.

Commentaires

Leave a Reply

Découvrez l’écosystème FL330

FL330, c’est toute l’aviation professionnelle — explorez nos autres espaces.