Every trade show has the same slide: a vibration trace, a red marker two weeks before the removal, a caption saying the failure was predicted. Nobody in the room disputes it, because there is nothing to dispute — the event is real and the algorithm did flag it. The slide simply answers a question no maintenance organisation has: whether the failure could be seen in data that already existed. The questions the operator actually has to answer are what the same detector sends on the several thousand aircraft-weeks in which nothing was wrong, who is allowed to read the data it feeds on, and what any of it is permitted to change in the approved maintenance programme.
The demonstration is the easiest part of the problem
A retrospective demonstration is built backwards. The event is known, the aircraft is known, the parameter that moved is known, and the detector is tuned until the marker lands where the story needs it. None of that is dishonest — it is how a feature is found in the first place. It is only worthless as evidence of operational value, because every choice in the chain was made with knowledge that the shop will not have on the morning the alert arrives.
Two mechanisms are worth naming, because they survive into serious-looking validation work. The first is selection: the sample is drawn from aircraft that failed, so the detector is only ever asked the easy half of the question. The second is leakage, where information about the outcome creeps into the inputs — a parameter that only exists because a troubleshooting task was already opened, a data window that quietly extends past the removal, a threshold set after looking at the test set. Leakage inflates apparent performance without anyone intending it, and it is invisible in a conference room.
The honest version of the demonstration is dull and much more expensive to produce: run the detector forward, on a fleet it has never seen, over a period long enough to contain both the failures and the far larger number of non-failures, and count everything it said. That is the only measurement that predicts what the shop will receive.
The slide at the trade show is not proof the system works. It only shows the system can spot a failure after it happens. The real test is seeing how often it flags real problems in normal use, and how often it gives false warnings.
A detector might flag 100 false alerts for every 10 real failures. If the fleet has 100 aircraft and the failure happens 20 times a year, the shop gets 100 false alerts and 18 real ones. That means 94% of alerts are wrong, even if the system is 90% accurate at spotting real failures.
Base rates decide what the shop actually receives
The reason a good detector can still flood a hangar has nothing to do with the quality of the model. It is arithmetic, and it depends on how rare the failure is. Consider a fleet of a hundred aircraft, a failure mode that occurs twenty times a year, and a detector evaluated once a week per aircraft. That is 5,200 opportunities to speak, of which twenty are real.
| Quantity | Detector A | Detector B |
|---|---|---|
| Opportunities per year (100 aircraft, weekly) | 5,200 | 5,200 |
| Real events in the period | 20 | 20 |
| Sensitivity (real events caught) | 90% — 18 caught | 90% — 18 caught |
| False alert rate on non-events | 5% | 0.35% |
| False alerts produced | 259 | 18 |
| Share of alerts that are real | about 6% | about 50% |
| Investigations per real finding | roughly 15 | 2 |
Both detectors catch nine failures in ten. Marketing material describes both as ninety per cent accurate. One of them asks the shop to open fifteen investigations to find one defect; the other asks for two. The difference between them is not sensitivity, which is what demonstrations show, but the false alert rate on the enormous population of healthy aircraft, which demonstrations almost never show because it requires the boring half of the dataset.
This is why the useful question at a stand is never does it detect. It is: over what period, on how many aircraft, how many alerts did it raise in total, and how many of those closed with a confirmed finding. If the answer is a percentage rather than two counts, it has not been measured in service.
The trap. A detector quoted at 95% accuracy on a rare event is usually worse than useless: a model that says nothing is wrong every single time would score above 99% on the same fleet. Accuracy is meaningless when the classes are unbalanced. Ask for alert counts and confirmed findings, never for a single headline percentage.
An alert is only useful inside a planning window
A prediction that arrives four days before failure is an AOG with extra paperwork. A prediction that arrives four months early removes a part with useful life left in it, which is exactly the waste that condition monitoring was supposed to end. The value of a prognostic sits inside a window bounded on one side by the time needed to obtain the part, a slot, a qualified team and, where relevant, the authority’s agreement, and on the other by the point at which acting early costs more than the unscheduled event would have.
That window is not a property of the algorithm. It is a property of the operator: spares policy, network shape, whether the aircraft night-stops at a station with capability, contractual turnaround commitments, how far ahead the maintenance plan is frozen. The same alert horizon that is comfortable for a hub-based long-haul fleet is useless for a regional operator whose aircraft only meets a capable station once a fortnight. A supplier who cannot express the lead time distribution of its alerts — not the mean, the spread — has not thought about the customer’s constraint.
The mirror question is degradation speed. Some failure modes progress slowly and predictably enough that a trend has meaning; others are dominated by a random initiating event, and no amount of history predicts them. Deciding which category a given failure mode belongs to is reliability engineering, not data science, and it is the step most often skipped.
What an alert is allowed to change
This is where enthusiasm meets the approved documents. An operator’s maintenance programme is an approved document. Under the European framework, tasks and intervals derive from the manufacturer’s maintenance planning data, itself built through the industry MSG-3 analysis and consolidated by the Maintenance Review Board, and the operator’s programme is approved by, or under a procedure agreed with, the competent authority. A vendor algorithm sits outside that chain until somebody puts it inside.
Practically, an alert can occupy one of several very different positions, and the contract should say which one:
- Advisory only. The alert produces an internal engineering notification. No approved task moves, no interval changes, and the aircraft remains serviceable under the existing programme. This is where almost every deployment starts, and there is nothing wrong with staying there.
- Trigger for an existing task. The alert causes an already-approved inspection or functional check to be performed earlier than its interval would require. Nothing new is invented; the timing changes. Straightforward, and the paperwork already exists.
- Credit inside the programme. The monitoring replaces or extends a scheduled task. This changes the approved programme and requires the manufacturer’s data and the authority’s agreement. It is also the only case in which the predictive system actually removes cost from the schedule rather than adding investigation load.
- Airworthiness-driven. Where a task exists because an airworthiness directive or a certification maintenance requirement demands it, monitoring does not substitute for it on the strength of a supplier’s confidence.
The distinction matters commercially. A system in the first two positions is a productivity tool whose benefit is measured in avoided disruption. Only the third reduces scheduled work, and only the third requires a regulatory case — which the supplier may or may not be prepared to build, and which is rarely part of the price quoted at a stand.
The right reflex. Before discussing price, write down which of the four positions above the alert will occupy in your organisation on day one, and who inside the CAMO signs when it fires. If nobody can name that person, the pilot project will end with a mailbox nobody reads.
Data access is a contractual question, not a technical one
Predictive maintenance runs on data that already exists and already has owners, custodians and legal constraints. Recorded flight data, condition monitoring parameters, transmitted health messages, technical log entries, component removal histories, shop findings: each has a different origin and a different set of rules attached. Bundling them into a supplier’s platform is a transfer of something valuable, and it deserves the same scrutiny as a spares agreement.
| Question | Why it decides the outcome | Where it must be settled |
|---|---|---|
| Which parameters leave the aircraft, at what rate, and by which link | Determines what the model can ever see, and the recurring transmission cost | Technical annex, with the scope frozen |
| Who may read raw data as opposed to derived indicators | Raw recorded data may identify a crew and a specific flight; indicators usually do not | Data protection clause and the operator’s flight data policy |
| Whether the supplier may reuse fleet data to improve models sold to others | Common, often silent, and sometimes acceptable — but it should be a decision | Licence and reuse clause |
| What comes back if the contract ends | Without portability, five years of history stays with the vendor and the operator restarts from zero | Exit clause, with format and deadline |
| Retention periods and deletion | Aligns with the operator’s own retention rules and legal obligations | Retention schedule |
| Whether alert logic can be audited or is a black box | Determines whether the authority can ever be shown a case for credit | Transparency clause |
The crew identification point deserves emphasis. Where the operator runs a flight data analysis programme, its entire legitimacy — under the ICAO framework and the operator’s safety management system — rests on non-punitive use and controlled de-identification. Routing the same data into a maintenance analytics platform with a different user population and a different contract can quietly break the protections that made the programme acceptable to the crew force in the first place. That is a conversation to have with the safety department and the crew representatives before the pilot project, not after somebody notices.
The incentive nobody puts on a slide
Ask who pays for a false alert. In many arrangements the answer is the operator: the investigation labour, the removal, the shop visit, the loaner, the schedule disruption. The supplier’s exposure is reputational. That asymmetry pushes the threshold in one direction — towards sensitivity, because a missed failure is visible and blamed, while a hundred quiet false alerts look like diligence.
The asymmetry sharpens when the same group sells the alert and the replacement part. Nobody needs to act in bad faith for the outcome to drift; a threshold set slightly conservatively, repeatedly, produces a removal rate that suits one party better than the other. The counterweight is measurement: a NFF rate tracked on removals triggered by the system, compared with the NFF rate on removals triggered the old way, and a mean time between unscheduled removals watched over the same period. If a predictive system is working, unscheduled removals should fall and the confirmed-finding share of alerts should rise. If removals rise and findings do not, the system is selling parts.
An availability-based contract can reverse the incentive, because the supplier then carries the cost of both the failure and the unnecessary removal. Where it is written that way, that alignment is worth more than any accuracy figure.
Running it in shadow before it runs the shop
The defensible way to introduce a predictive system is to let it speak into a void for long enough to be counted. In a shadow deployment the alerts are recorded, timestamped and closed against what actually happened, but they do not trigger maintenance action. It costs a subscription and some engineering attention, and it produces the two numbers that no demonstration can supply: how many alerts per aircraft per month, and what share of them corresponded to a finding.
A few design points make the shadow phase worth the delay. Fix the thresholds before the period starts and resist adjusting them mid-run, or the result is another retrospectively tuned demonstration. Include the aircraft that are least representative — the oldest airframes, the different engine standard, the ones based where the environment is hard — because heterogeneity is where fleet-wide models break. Log the non-alerts too, so that a failure the system stayed silent about is counted rather than forgotten. And agree in advance the number that would make the deployment a failure, because a threshold defined afterwards is always met.
Operators comparing notes on how these deployments actually behaved in service will find more candour in a peer discussion than in a vendor case study; the FL330 forums carry that kind of exchange between maintenance and engineering staff.
What to ask before leaving the stand
| Question | Answer that means nothing | Answer worth following up |
|---|---|---|
| How many alerts did the system raise last year across your customer base? | Our detection rate is above 90% | A count, a fleet size and a period, with the confirmed-finding share |
| What is the lead time between alert and failure? | Weeks in advance | A distribution, including the short tail and the very early alerts |
| Has any authority accepted this monitoring as programme credit? | We are working with the regulators | A named case, or a clear statement that it is advisory only |
| What happens to our data if we leave? | Your data is yours | A format, a deadline and a deletion commitment in the exit clause |
| Who bears the cost of an unconfirmed removal? | False alerts are rare | A cost-sharing mechanism, or an availability contract |
| Can we see the alert logic? | It is proprietary | Documented features, thresholds and a change log |
None of this argues against predictive maintenance. Condition monitoring displaced fixed intervals for much of an engine’s content long ago, and it works. It argues that the thing being bought is not detection — detection is the easy part and it was demonstrated years ago. What is being bought is a false alert rate the shop can absorb, a lead time that fits the network, a defined position inside an approved programme, and a set of data rights that survive the end of the contract. All four are contractual, and none of them is visible on the slide with the red marker.


Leave a Reply
You must be logged in to post a comment.