Internal audit before a regulatory visit: targeting what will actually be checked

Faisceau de lampe balayant les circuits hydrauliques dans un logement de train d'atterrissage sombre

The letter announces the visit for three months out. Compliance monitoring launches a full audit: six domains, roughly forty files, final report with no findings. On the day itself, the inspector opens four findings in two days, two of them in domains that audit had just covered. Nothing changed in the meantime: the sample had been drawn where there was nothing to find, and the empty report became the first exhibit in the file.

The first document requested is not the operations manual

Inspectors arriving for a scheduled visit rarely start with the manual. He asks for the internal audit programme, the findings register, the dates actually carried out against the dates planned, the closure deadlines. Within an hour he knows what the organisation is not looking at.

This is the underlying shift in modern oversight. Unable to check the product file by file, the authority checks the function that, internally, is supposed to do so: that component of the management system, independent of the activity it controls, equipped with a programme covering every domain over a defined cycle. On the American side the vocabulary differs and the logic is identical: oversight is organised around system elements and their attributes — accountability, authority, procedures, controls, process measurement, interfaces.

The underlying question is the same on both sides: does this organisation find its own non-conformities before a third party finds them for it? Hence a counter-intuitive consequence: a register full of substantive non-conformities and documented closures is reassuring, an empty register is worrying — a detection system that detects nothing is, statistically, broken.

Inspectors don’t check every file. They check if your internal audit system actually finds problems. If your audit finds nothing, it’s a red flag – it probably means your system isn’t working.

A regional airline with 50 aircraft and 200 flights daily had an internal audit that reviewed 40 maintenance files and found zero non-conformities. During the regulatory visit, inspectors found 3 serious issues in deferred maintenance logs – all in files not reviewed by internal audit. The audit had sampled only routine flights, missing the exception cases where problems actually occurred.

An audit that finds nothing measures, above all, its own power

The calculation is elementary and almost never done. If a given proportion of files carries the non-conformity, the probability of catching at least one in a random draw equals one minus the probability of missing all of them. With two non-conforming files in a hundred, ten files examined give less than a one-in-five chance, thirty give a little less than a fifty-fifty chance. An audit that reviewed forty files without finding anything has not demonstrated that the population is sound: it has demonstrated that it lacked the power to know.

The inspector’s question follows from this, formidable because it is simple: how did you build your sample? The answer “at random, from the year’s files” is itself the real finding: the audit was sized to be completed, not to find anything.

There is something worse than low power: poor targeting. Non-conformities are not evenly distributed; they concentrate in the areas that escape routine. Drawing flat across the whole population compounds both defects.

Probability of finding at least one non-conformity by sample sizeChart. On the x-axis, the number of files examined, from zero to sixty. On the y-axis, the probability of hitting at least one non-conforming file, from zero to one hundred percent. The solid curve corresponds to a population containing two percent non-conforming files: it climbs slowly and does not reach half at thirty files. The dashed curve corresponds to ten percent non-conforming: it passes three-quarters before fifteen files, then saturates. A point marks thirty files on the solid curve.0204060files examined0255075100probability of finding a non-conformity (%)10% non-conforming2% non-conformingless than a fifty-fifty chance
On an almost sound population, a small sample proves nothing. At two percent non-conforming, it takes dozens of files for a fifty-fifty chance of hitting the non-conformity; at ten percent, fifteen are enough.

The trap: the dry-run audit launched three weeks before the visit. It leaves findings open with no root-cause analysis and no deadline held. A non-conformity open for four months, with a dated action plan and a first effectiveness measurement, is a strong point; the same non-conformity opened the day before is an admission.

The geography of non-conformities

Four families of situations recur most often in what a visit uncovers, and all of them fall outside the scope of an audit organised domain by domain.

Interfaces. Between the operator and the continuing airworthiness management organisation, between that organisation and the workshop, between the operator and the ground handling agent. Each party audits its own side of the bridge, no one audits the deck: that is where technical logs, load handovers and qualification updates get lost.

Exceptions. The item deferred under the minimum equipment list, carried over once then twice; the non-scheduled flight, the ferry flight after maintenance; the waiver granted one evening by phone. The normal procedure is written, learned, checked; the exception is not, and that is exactly what an inspector asks to see.

Interim postings. A nominated person absent, a deputy acting without the deputisation having been formalised. Verbal delegations leave no trace, and a signature affixed without the corresponding authority contaminates every act of the period.

Transitions. New manual revision, new flight-planning tool, new type added to the certificate. A transition opens a window where two versions coexist. It closes; the files produced while it was open remain.

Where internal audit settles in, where oversight samplesFour-quadrant matrix. On the x-axis, the frequency of the activity, from routine to rare. On the y-axis, the density of traceable records, well documented at the bottom to poorly documented at the top. Bottom left, routine and well documented: regular flights from the main base, where internal audit settles in. Bottom right, rare but documented: ferry flights and test flights. Top left, routine but poorly documented: briefings and verbal instructions. Top right, shaded cell: interim postings, waivers, deferred items, subcontracting, isolated outstations — the blind spot, the one oversight samples first.record densitypoorly documentedwell documentedroutinefrequency of the activityrare, exceptionalblind spotinterim posting, waiver, deferralsubcontracting, isolated outstationbriefings and verbal instructionsunwritten practicesregular flights, main baseinternal audit settles hereferry flights, test flightsrare but documented
Routine audit and oversight do not target the same cell. The comfort zone of an audit is bottom left; non-conformities sit top right. Every campaign should know how many of its draws landed in the shaded cell.

Sampling like the inspector: starting from the data, not the calendar

The programme must cover every domain over a defined cycle: that is a requirement. But covering everything does not mean spreading the effort evenly. The cycle gives the minimum coverage; the organisation’s own data says where to put the rest.

Two men review financial data on a computer, preparing for an upcoming regulatory inspection.
Auditors scrutinize financial records to ensure compliance before regulators arrive.

This data already exists and is rarely used for this purpose: occurrence reports, technical delays, findings from ramp inspections abroad, non-conformities flagged by ground handling agents, line-check results, borderline cases in flight and duty time, deferred maintenance due dates. Each stream identifies a stratum. Hot strata get over-sampled and a light draw is kept on the rest: for the same effort, the probability of hitting a real non-conformity rises sharply, since the effort shifts toward the areas where the proportion of non-conforming cases is highest.

What remains is the technique that yields the most, and that inspectors routinely use: cross-tracing. Instead of examining fifty files of the same kind, take one real flight — preferably a night flight or a diverted outstation — and trace its entire chain: crew qualified, current on all checks, trained on the manual revision in force that day; minimum equipment list applied and the time limit correctly counted down; flight file, fuel, weight and balance and weather consistent with what was actually done. A single flight crosses every domain and puts the interfaces under tension.

It must still be acknowledged that the map of hot strata is not the same from one operation to another. A scheduled operator on a dense network concentrates its risk in repetition: a practice drifting slowly, a tolerance turned local norm at one outstation, a non-conformity spreading because everyone does the same thing. An on-demand operator, by contrast, lives almost entirely in the exception — a destination never served before, a crew put together the night before, unfamiliar handling, a decision made outside office hours; for them the blind spot is the normal regime, and flat sampling loses all meaning. A recently expanded fleet, or a manual under revision, adds the stratum of transitions; heavily subcontracted operations add that of contractual interfaces. A defensible sampling plan comes down to one sentence: what varies the most in this operation, and why that is where the effort belongs.

The right instinct: the rule of three threads. Before each campaign, pick three real flights in three strata — an exception, an interface, a transition — and trace each one back to the archived record. What you get from this cannot be obtained any other way.

Two ways to build a sample, at comparable effort
Criterion Flat draw Data-driven draw
Starting point the programme calendar recent internal signals
Population targeted all files in the domain the strata designated hot
Unit examined the document, in isolation the flight or the deadline, end to end
What it finds frequent, minor errors interface breakdowns
What it misses the exception, the interim posting, the subcontractor the slow drift of the normal regime
Effect on the visit the inspector redoes the work the inspector checks yours

What is not recorded is not compliant

An inspector does not record what is true, he records what is demonstrable. “We do it, it’s just not written down” does not score as compliant: an untraced practice survives neither the departure of the person who carried it nor a challenge.

Two objects that documentation conflates deserve to be kept separate. The procedure states what must be done: that is the obligation. The record attests that a specific act was carried out, by an identified person, on a known date, on the basis of an identified version of the text.

A useful record is attributable — you know who, when, on which version —, readable without its author, and retrievable within a time compatible with its use. Migrating an IT system, moving to electronic signature and outsourcing archiving all silently erase years of records.

One question deserves to be asked in cold blood, because it will be asked in the heat of the moment: on which version of the procedure was this person trained, and where is the proof that they received the following amendment? Many distribution systems keep proof that something was sent; actual acknowledgment of receipt and the exact revision index it relates to are far less often kept.

The right instinct: the ten-minute test. Pick three records of different kinds — a training file, a technical log entry, proof of amendment distribution — and ask someone other than their usual custodian to produce them. What does not come out will not come out in front of the inspector either.

The internal finding: what makes it credible

Opening a finding costs nothing; closing it properly is a craft. Three notions are constantly conflated, and oversight separates them systematically.

Three possible responses to a finding, and what each one demonstrates
Response What it does What it demonstrates What it does not demonstrate
Correction brings the case back into compliance that the non-conformity was dealt with that it will not recur
Corrective action acts on the cause: procedure, interface, workload that the cause was addressed that behaviour has changed
Effectiveness verification re-samples the same stratum after a set period that the result has changed that other strata are sound

Root cause is the point where most files come unstuck. “Agent error” is not a cause, it is a description of the non-conformity. The cause is what allowed the error through: a procedure impractical as written, a workload that makes the check theoretical, an interface where each side assumes the other is checking.

Then the deadline. A long deadline that is held is worth more than a short one pushed back twice: slippage measures the gap between what the organisation commits to doing and what it is actually able to do.

Finally the effectiveness verification, and this is where the loop breaks almost everywhere. Amending the procedure and issuing a notice are not proof of effectiveness, they are acts. Effectiveness is demonstrated by going back, after a set period, and sampling the same stratum again. A finding reopened after closure costs more than a finding left open: the loop does not close.

The path of an internal finding, and where the loop breaksFlow diagram. An open finding leads to a question: is the root cause identified? If yes, corrective action, then effectiveness verification by re-sampling; closure only follows if the new sample no longer shows the non-conformity. If no, a mere correction with no lasting effect, and a dashed arrow leads back to the finding: the non-conformity returns. A note flags that most systems stop before the verification step.finding openroot causeidentified?corrective actionverificationby re-samplingclosurecorrection with no lasting effectyesnonon-conformity absentthe non-conformity returnsmost stop here
The bottom branch is not a methodological error, it is the default path. Without root-cause identification, the non-conformity resurfaces in a different guise. The solid line is the only path that allows for a closure that will hold up.

The trap: closure by paperwork. “Procedure amended, notice issued, finding closed” proves that a text changed, not that behaviour changed. The inspector will go and check for himself, on the ground, what the organisation did not check.

The two days: what plays out in the room

Material preparation matters more than people think. A single point of access to records, a domain-specific contact person who knows where to look rather than an executive who knows how to answer: this avoids the waiting periods during which an idle inspector widens his sample.

One rule tolerates no exception: never produce a record created for the occasion. A file’s metadata, the ink of a signature, the uniformity of a binder tell their own story. A record reconstructed after the fact no longer raises a question of compliance but a question of integrity — which is not scored at the same level, nor fixed with an action plan.

Answering the question asked is the second rule: producing three documents when only one was requested opens up one more avenue. Nominated persons, for their part, are tested on one specific point: do they know their own documentation? The most productive question is not “what do you do in that case” but “show me where it’s written.” For subcontracted activities, responsibility cannot be delegated: the inspector will ask for the contract and what it provides for the authority’s right of access to the provider’s data, that provider’s last audit, and what became of its findings.

What to go and check in your own documentation

Six questions can be checked in an evening. Do the audit dates actually carried out coincide with the dates planned in the announced cycle? Is the audit function independent of what it audits? How many substantive internal findings over twelve months, and in which domains — a domain with no findings at all points to a blind spot, not to excellence? What is the median closure time, how many deadlines have been pushed back? How many closures rest on an effectiveness verification? Do findings coming from outside feed into the same register as internal ones?

A grey area remains, and it is better to address it than to have it come up in the room: independence in a small organisation. Combining functions is generally accepted there, self-auditing is not, and the boundary between the two is written down precisely nowhere. It is demonstrated by bringing in an external auditor wherever combining roles is unavoidable.

For anyone preparing to take up a nominated-person or compliance-manager role, the point to work on is not reciting the manual, but the ability to explain how samples are chosen, and why that particular choice.

This article describes mechanisms; it does not set out any obligation. The applicable regulation, the means of compliance published by the authority, and the operator’s approved or accepted documentation prevail in all circumstances over what is written here.

Questions fréquentes

Must every domain be audited every year?

The programme must cover every domain over a cycle defined by the operator and known to the authority; coverage is an obligation, the distribution of effort within the cycle is not. Nothing prevents a minimal draw on stable domains and concentrated depth on the strata the data points to. What gets checked at the visit is whether the announced cycle was actually followed and whether gaps between planned and actual dates are explained.

Is a domain with no findings at all for two years a problem?

Not in itself, but it is an anomaly that calls for an explanation. On real activity, a large enough sample eventually turns up minor non-conformities; their prolonged absence more often signals a sample that is too small or poorly targeted than operational perfection. The expected answer is the detail of the sampling plan, not a statement of compliance.

What is a dry-run audit by an outside party, done just before the visit, actually worth?

It has value if it is carried out early enough for its findings to be analysed, addressed and verified before the visit. Launched three weeks beforehand, it mainly produces open findings with no root-cause analysis, which the inspector will find and read as an admission. The outside party is most useful where internal independence is hard to demonstrate, particularly in small organisations.

Can an open internal finding worsen the operator's position with the authority?

A finding that is properly analysed, addressed within a self-imposed deadline and verified by re-sampling works in the organisation’s favour, because it demonstrates that the detection function works. What makes things worse is a finding left open with no deadline, a deadline pushed back several times, or a closure with no proof of effectiveness. What makes things worse still is a non-conformity the authority itself raises when it was already sitting, untreated, in the internal register.

How can the effectiveness of a corrective action be verified without waiting months?

By defining, as soon as the finding is opened, the stratum to be re-sampled, the sample size and the date of the check. On frequent activity, a few weeks are enough to build up a usable population; on rare activity, a long deadline has to be accepted and written down, rather than closing the finding with a documentary amendment. Writing the verification step in at the moment of opening is also what prevents a purely declarative closure.

What can the inspector ask about a subcontracted activity?

Since responsibility remains with the operator, the inspector can ask for the contract and what it provides for the authority’s right of access to the provider’s premises and data, that provider’s audit programme and audit reports, and how the findings from them were handled. Applicable provisions generally also allow the authority to visit the provider directly. A subcontracted activity with no audit of its own is a system-level non-conformity, regardless of the quality of the service delivered.

Commentaires

Leave a Reply

Découvrez l’écosystème FL330

FL330, c’est toute l’aviation professionnelle — explorez nos autres espaces.