Professional References or a Written Test: Which One Actually Predicts Line Success

Bureau sobre avec chemise fermée, téléphone d’acier et bloc-notes vierge dans une pièce bleu nuit

Two files land on the same morning: the first shows a high written-test score and lukewarm references; the second, exactly the reverse. Procedure does not say what to do with this contradiction — it says to add them up. What remains to be known is what each of the two sources actually tells us about what comes next: adaptation on the line, first solo release, and whether standards hold three years on.

Before comparing predictors, write down the criterion

A predictor has no validity in itself: it has validity relative to a criterion, a quantity measured later that has been decided to represent success. As long as that criterion is not written down, the comparison spins in a void: everyone defends the source that predicts well whatever they already have in mind, without saying what that is.

Three criteria circulate in training departments. The first is binary — is the qualification obtained? — and almost constant: a course brings already-filtered candidates up to standard, and one cannot predict a variable that does not vary. The second is far more informative: the cost of adaptation, in extra sessions, in extension, in a retaken check. The third concerns safety and is the hardest to measure: how standards hold over time, out of sight.

There remains the question that is always skipped: is this criterion reliable? Two instructors grading the same session do not reach the same appraisal, and the gap between them can exceed the one separating two candidates. A noisy criterion caps any relationship: the end measure sets the maximum attainable, whatever the quality of the initial test. The first useful expenditure is therefore not a better test, but a behavioural grid and calibrated evaluators.

A predictor (like a test or reference) only tells you something useful if you know what you’re measuring (the “criterion”). If your measure of success is unclear or inconsistent, the predictor can’t help you. First, define what “success” really means in your case.

An airline might define “success” as how quickly a new pilot adapts to their procedures. If two check pilots rate the same new pilot differently by 20%, that inconsistency makes it impossible to know if the written test or references are actually predicting anything useful.

The written test: reliable, transparent, narrow

A knowledge test measures two things at once, rarely the one people assume: declarative stock — regulations, systems, performance — and the ability to process a question quickly under time pressure. It is the second that carries most of the link with learning, which consists of absorbing an unusual volume of information within a few weeks.

Its strength is reliability: the same candidate, on a parallel form, gets a similar score. That is also its trap, because reliability gets confused with validity — an instrument can measure the wrong thing in a perfectly stable way. It is also crammable, and cramming does not add neutral noise: it pushes upward only the candidates who had access to the question bank, and the score stops separating by what people know and starts separating by what they managed to obtain. What remains are non-predictive virtues that carry real weight: a dated score justifies a refusal in front of a candidate or an authority; an impression does not.

The trap: a circulating question bank does not show up in the average, but in the shape of the distribution. If scores bunch up against the maximum and the most discriminating items are passed by almost everyone, the test has stopped sorting candidates while keeping the authority of a number. Track the spread, not the average.

The reference: the right material, a poor channel

A reference covers what no test can reach: behaviour observed over time, often by someone who was not there to assess. Adherence to procedures out of sight, reaction to a remark, attitude within a crew when the day runs long. It is the material closest to the criterion.

The problem is not the material, it is the channel. The candidate chooses their referees, and no one names the one who would say no. The referee speaks under constraint, in a small world where careers keep crossing paths. The scale is compressed at the top — everyone is « very good » — so the useful information sits in the nuances and the silences. A global judgment finally colours every answer, including on areas never observed.

A good part of this can be recovered through the shape of the interview. « Would you recommend them? » invites a social answer; « how many times in the last twelve months did you have to bring up a procedural point with them? » invites a fact. Question the person who trained or checked them, not their course-mate. And a former employer who confirms only the dates and nothing else is not delivering a lukewarm reference: they are leaving a blank box, which nothing entitles you to read as a green light.

The right reflex: a template of seven to ten closed questions about dated facts, filled in during the call and signed by whoever made it. It makes candidates comparable, survives the recruiter’s departure, and turns a statement into a piece of the file.

Two sources, two error profiles: the « misses » rows do not overlap.
Comparison Written test References
What is measured Declarative stock and ability to process a question quickly Behaviour observed over time, filtered by a third party
Reliability High, verifiable on a parallel form Low: depends on the referee and the relationship
Dominant bias Unequal access to prep material, reading speed Leniency, global judgment, cautious silence
Best predicts The cost of the learning phase How standards hold over time and crew performance
Misses The candidate who can do the job but reads poorly under the clock The candidate who is likeable, and the one whose former operator stays silent
Justifying a refusal Good: the score is dated and can be relied upon Weak if the call was not recorded

Why the correlation always seems to vanish

An operator who compares, among its serving pilots, the entry score against how adaptation went almost always finds a weak relationship, and draws the conclusion that suits it: only judgment matters. This is an artefact. The follow-on is only observed among those who were hired, i.e. the upper part of the predictor: the true relationship holds across the full range of candidates, while the one being measured covers only the retained slice, where scores are compressed and chance weighs proportionally more. The slope flattens mechanically: concluding that the predictor predicts nothing amounts to measuring a slope after cutting off the axis.

Effect of range restriction on the observed slopeScatter plot linking the test score, on the x-axis, to performance measured later, on the y-axis. Across the full range of candidates the relationship rises clearly, shown as a solid line. A dashed vertical marks the hiring threshold; to its right, among the retained candidates only, it becomes nearly flat, shown as a dashed line.full rangeafter selectionmeasured performancetest scorehiring thresholdretained population
A slope can vanish by construction. By observing only the retained candidates, the relationship is measured on the upper part of the axis alone: it collapses. The predictor has not changed — the sample has.

The remedy costs little: keep, within the limits of data protection, a record of decisions and their outcomes, and track in particular those who were hired despite a low score — a waiver, recruitment pressure, a second attempt. They alone provide information on the lower part of the axis: whoever never hires below its threshold will never know whether that threshold sits in the right place.

The useful question: what the second source adds

« Test or references » is a poorly framed question. A source is not assessed on its own, but against what already sits in the file: a mediocre source whose errors don’t resemble those of the first adds more than an excellent but redundant one.

Overlap between what the written test explains and what references explainTwo circles overlap partially. The one on the left, outlined with a solid line, represents the share of success explained by the written test; the one on the right, outlined with a dashed line, the share explained by references. The shared area is narrow: most of each circle remains specific to its source, and it is this unshared crescent that constitutes the marginal contribution of the second source.written testreferencesown contributionown contributionredundant information
A second source is worth only what it explains that the first does not. Two highly correlated sources reinforce each other without adding anything; two sources whose errors differ each catch the cases the other misses.

Yet the two get it wrong in almost mirror fashion. The test misses the candidate who can do the job but reads a question poorly against the clock, and lets through the one who has crammed the question bank. The reference misses the candidate who is likeable, and stays silent on the one whose former operator applies a policy of silence. The cases missed are not the same ones: that is the only serious reason to pay for both.

The practical consequence is surprising: a reference should not be used to confirm what the score already says — that is comfort, not information. It should be called upon where the score is silent — the middle zone, where a few points’ difference means nothing, and the suspicious extremes.

What a contradiction between the two sources should trigger: an action, not an average.
State of the references Score above the threshold Score below
Supported by dated facts Simple decision; just check the session was not sat on compromised items Maximum information: a targeted second measure, not a debate about the threshold
Blank or channel unavailable A blank is not a green light: use a replacement channel and closely spaced check-ins The best-founded refusal: no source supports the opposite
Qualified on a specific fact The fact takes precedence: establish its nature, date and consequences before weighting it A threshold decision, not an average one: this type of reservation cannot be offset by points

Weighting without lying to yourself

Weights are set before seeing the files, and they are written down. A weight decided after reading is not a weight, it is a justification: the coefficient adjusts, in perfectly good faith, until it lands back on the candidate already chosen. The order of operations matters more than the value of the coefficients.

A man and a woman are sitting at a table in an office, both writing on papers, with a whiteboard and stacks of documents in the background.
Written tests assess skills objectively, while references may reflect bias.

Second rule, a more demanding one: write down the prediction. One line per candidate hired — score, reference appraisal coded on a short scale, adaptation forecast in two or three classes — then, twelve to eighteen months later, the line is reread and checked against the facts. This is the only operation that turns collective opinion into local knowledge, and the one almost never done, because it exposes the person who does it.

Third rule: do not pour everything into an average. The compensatory model suits aptitudes, where a point lost here is recovered there; it suits neither integrity nor safety, since a deliberate breach of a procedure is not offset by a good systems score. These dimensions must be handled by threshold, written into the procedure, otherwise it will be negotiated away whenever recruitment pressure is highest. The test’s threshold, for its part, carries two costs: fewer long adaptations, but rejected candidates who would have succeeded — and this second cost is never seen.

One last parameter escapes the coefficients: time. Neither source keeps its separating power indefinitely, and they do not decay at the same rate — which rules out setting a weighting once and for all and reusing it for a candidate whose measurement is old.

Lifespan of each source’s predictive powerTwo curves plotted against time elapsed since hiring. The solid line, the written test score, starts high and drops sharply. The dashed line, behaviour reported by references, starts lower and declines slowly. The two cross at a point marked with a marker; beyond it, the second source separates profiles better. Qualitative y-axis, no values.test scoreobserved behaviourcrossoverpredictive powertime since hiring
Predictive power has a lifespan. Qualitative scale: what matters is the crossover. The knowledge measured ends up being acquired by everyone, whereas what references describe keeps on separating profiles.

This shape has a direct consequence: a score two years old no longer carries the weight it once did, since the knowledge measured has either been acquired by everyone or lost. Decide on a validity period and write it down. Interview templates, meanwhile, circulate faster among practitioners than in manuals, on the FL330 forum as much as in training committees.

What the type of operation changes

A weighting is not transferable, because the criterion itself changes with the operation. In scheduled transport, adaptation is long, standardised and expensive: the most useful criterion is the cost of the learning phase, and that is where the written test finds its most solid footing. In business aviation, where the crew deals with a programme changed the day before, assistance that varies from one airfield to the next, and sometimes a principal on board, the useful criterion is holding up out of sight and the ability to raise a well-argued refusal: the informative source is the one that describes behaviour, not the one that measures a stock of knowledge.

Aerial work and freight shift the cursor further still — night flights, poorly equipped airfields, irregular rhythms. The question put to the referee is no longer « does he know his systems » but « what does he do when the day runs long and no one is watching ». At a training organisation recruiting instructors, the criterion shifts a third time: what needs to be predicted is no longer piloting but the ability to bring a student on and to render a defensible judgment. Neither the knowledge score nor the operational reference speaks seriously to this; only an observed session does, and its absence explains a good part of the instructor hires that disappoint.

Hence a common mistake among organisations writing their first procedure: taking a larger operator’s grid as it stands imports coefficients calibrated on a different criterion and a different pool. The skeleton can be copied — defined sources, order of operations, traceability; the weights cannot.

What the framework already requires

Regulation does not say how to select; it imposes checkpoints. The operator ensures licences, ratings and recent experience are in order, conducts adaptation training and checks, and keeps records of them. Depending on scope, these are joined by background verification for security purposes and, at the operators concerned, a psychological evaluation of the pilot before entry into line service. None of this is a selection tool, but together they occupy a share of the ground that references used to cover alone.

One development deserves the attention of anyone recruiting internationally: the requirement placed on a US carrier, under the conditions set by the applicable federal text, to consult before hiring a pilot’s employment, training and check history in a registry fed by successive employers. The difference from a phone call is not one of content but of nature: a dated fact cannot be retracted. In Europe, the method remains up to the operator, under two constraints: collect only what serves the decision, and write about a person only accurate elements that are accessible to them.

The trap: what you write down during a reference call becomes part of the person’s file. A conditional-tense appraisal, a judgment reported without a source, a remark about health slipped in by a former manager: all of this can be requested by the person concerned and will have to be justified. Record facts, their date, and their author.

The three zones that remain grey

The first is the absence of a channel: operator gone out of business, a military career, a long break, no one to call. The mistake is to read this blank as a doubt and to silently mark the candidate down. The procedure must provide a replacement path — an evaluation session, a probationary period with closely spaced check-ins — otherwise everyone will improvise their own, always to the detriment of the same profiles.

The second is the weight of time. An unfavourable fact five years old, followed by a clean record, does not carry the same weight as the same fact occurring last year, but no text says how fast it should fade. Write your own rule, even a rough one.

The third is a change in standard. Coming from an organisation where procedure was oral into one where it is written, a candidate is judged by their referees against a frame of reference that is not yours. Asking the referee what they are judging against often changes how a reservation should be read.

For anyone preparing a file, the consequence is simple and rarely followed: check who will speak for you. An instructor who checked you recently speaks of dated facts; a former colleague speaks about you. Also ask your operator what your training file contains: it is that record, not your CV, that will be consulted the day a recruiter wants a trace — FL330 courses prepare the score, not that trace.

This article describes measurement mechanisms, not an obligation. In the event of a discrepancy, the applicable text, the authority’s decisions, and the operator’s approved documentation — including the operations manual and internal procedures — prevail over everything above.

Questions fréquentes

Does a written-test score still hold value two years later?

It loses its informative power as the knowledge measured is either acquired by everyone or has become outdated. An old score mainly tells you about general aptitude, which changes little, and very little about current knowledge. The most honest approach is to set an explicit validity period in the procedure and require a parallel form to be retaken beyond it.

Should you call the former line manager or the instructor who checked the candidate?

The instructor or examiner first: they observed situations comparable to the ones the candidate will face with you, and they speak about dated facts. The line manager better informs on administrative reliability, availability, and team behaviour. The two calls do not cover the same ground and should not be blended into a single appraisal.

What should be done when a former employer confirms only the employment dates?

Treat the answer as missing data, never as a neutral or favourable reference. The absence of unfavourable information is not favourable information. The procedure must provide a replacement channel, for example an evaluation session or a probationary period with closely spaced check-ins, applied the same way to all candidates concerned.

Should the candidate be told in advance what questions will be put to their referees?

Yes, and announcing it does not weaken the process: the questions concern facts held by a third party, which the candidate cannot rewrite in advance. Advance information — who will be called, on which topics, what will be kept and for how long — is a matter of fair collection and fits in a few lines in the invitation letter. It has a useful side effect: a forewarned candidate proposes referees who are reachable and relevant, which reduces the number of files stalled for lack of a channel.

Can you check the validity of your selection process without a statistician?

Yes, provided you write down the forecasts beforehand and review them afterward. One line per candidate hired, a forecast in two or three classes, a review twelve to eighteen months later: the tally of correct and incorrect forecasts, compared with what chance alone would give, already tells you the essential. The difficulty is not technical — it lies in keeping the record and accepting the result.

Since the candidate chooses their own referees, does the reference still mean anything?

They will always choose them, and that is precisely why the content of the questions matters more than the identity of the referee. Frequency- and context-based questions, identical for every candidate, reduce the advantage of an obliging referee. Systematically asking to speak with whoever conducted the last check, rather than leaving the list entirely open, limits the effect further.

Commentaires

Leave a Reply

Découvrez l’écosystème FL330

FL330, c’est toute l’aviation professionnelle — explorez nos autres espaces.