Psychometric selection tests: why a good score can still be turned down

A candidate takes a psychometric test in a quiet, dimly lit room with other test-takers in the background.

The candidate left the room feeling they had kept up the pace: the multitasking section went well, working memory too, and the verbal feedback spoke of a result well above average. Three weeks later the answer is negative and explains nothing. The misreading is almost always the same: a score is read as a mark, when it is only a position within a group, set against a profile written before the test and cut into thresholds that do not compensate for one another.

A score is not a mark, it is a position

A selection battery produces no absolute mark. It produces a raw score — number of correct answers, mean response time, deviation from the instruction, number of omissions — that nobody uses as it stands. That raw figure is converted into a normed score: a percentile, a rating on a ten-class scale, a standard score centred on a conventional mean. The conversion says one thing only, and that is already a great deal: where this candidate stands relative to a reference group. Change the group and the raw figure does not move, while the reported result changes completely.

Three norm sets are commonly in circulation. The first compares the candidate with the general adult population, and it flatters just about anyone who has turned up for an aviation selection. The second compares them with everyone who has already sat that same selection: the pool is self-selected, often prepared, and the same raw score becomes median there. The third compares them with pilots already qualified and flying the line, a group more tightly grouped towards the top still. Verbal feedback announcing “above average” without naming the norm therefore says very little.

Two parameters sit behind the norm and sometimes weigh as much as it does: the age of the norming — a norm built long ago on a different medium, describing a group that no longer exists — and the conditions of administration: language of the test, equipment, order of the subtests. The battery has to be re-validated whenever one of these changes, an operation that is invisible to the candidate.

A psychometric test doesn’t give a simple pass/fail grade. It compares your performance to a group of other people, and that comparison can change completely depending on who that group is. A “good” score in one group might be average in another.

An A320 pilot candidate scores 85 on a psychomotor test. Against the general population, this puts them in the 95th percentile. But against other candidates who have taken the same test (often prepared candidates), their percentile drops to 65th. Against active pilots, it might only be 40th. The raw score is the same, but the interpretation changes completely.

One and the same raw score read against two reference groupsTwo bell-shaped distributions share the same raw-score axis. The left-hand one, drawn as a solid line, represents a general population. The right-hand one, dashed, represents all the candidates who have already sat this selection: its centre is markedly further to the right. A dashed vertical line marks a single raw score. Under the first curve, the area beyond that line is no more than a narrow tail; under the second, it covers a large share of the group. The raw score has not moved, the rank has.general populationother candidatessame raw scorehigh rankaverage rankraw score
The raw score does not change, the norm does. The same numerical performance reads as a tail of the distribution against a broad group, and as a median result against the selection’s real applicant pool. Asking which norm was used is worth more than arguing about the percentile obtained.

The target is a band, not a ceiling

Before the first test session, somebody wrote down what the job requires. This job analysis lists dimensions — processing information under time pressure, working memory, resistance to interference, but also rule awareness, cooperation, tolerance of uncertainty — and attaches to each of them not a maximum to be reached, but an expected range. This is the point most candidates miss, because it contradicts their entire school experience, where higher is always better.

On the cognitive dimensions the relationship remains roughly monotonic: fast and accurate beats slow and accurate. On the behavioural dimensions it no longer is. High assertiveness supports decision-making up to the point where it silences the other pilot’s contribution. A high concern for detail serves procedural rigour up to the point where it locks the crew onto a minor anomaly while the flight path drifts. Each of these scales has a bottom and a top, and the top is disqualifying for the very human-factors reasons that led to the writing of the crew competency frameworks.

Personality questionnaires additionally carry control scales: consistency of answers across equivalent items, tendency to present oneself in a favourable light, systematic extreme responses. These do not measure a quality, they measure how credible the rest of the protocol is. A perfectly smooth profile, high everywhere, matching the idea people have of the ideal pilot, is the most ordinary signal that the questionnaire was filled in as an exercise in communication.

Aviation selection isn’t about being the best at everything. It’s about being in the right range on all important skills. Too much of a “good” trait (like being overly assertive) can be just as bad as too little.

A candidate for a Boeing 737 first officer position scores 95% on assertiveness but only 40% on workload management. While their assertiveness score is impressive, their low workload management score means they wouldn’t be selected, as they might dominate the cockpit but fail to manage critical flight tasks under pressure.

The band expected by the job and two profiles that leave itSix dimensions along the horizontal axis, level on the vertical axis. A horizontal band represents the range expected for the job. A first profile, drawn as a solid line, stays inside the band on all six dimensions. A second, dashed, goes well above the top of the band on assertiveness and falls clearly below the bottom on workload management. None of its high points makes up for its low point.profile Aprofile Boutside the bandbelow the thresholdexpected bandlevelrulesarithmeticassertivenessspeedworkloadteamwork
A dimension is not a scale on which more is better. The profile sought is a range, bounded above as well as below. The dashed profile impresses on one dimension and leaves the band on another: it is the exit through the bottom that decides.

The trap. Answering the personality questionnaire “the way a good pilot would answer” produces precisely the profile that the validity scales are designed to spot: too favourable, too consistent, without a rough edge. The application is then set aside not on one dimension but on the credibility of the whole, and that reason is not put right by sitting the battery again in the same way.

Adding up or clearing the bar: two arithmetics, two verdicts

Once the scores have been normed, they have to be aggregated. Three logics coexist and do not give the same answer on the same application. The compensatory model computes a weighted sum: a very high peak buys back a dip elsewhere, and this is the one candidates spontaneously imagine. The conjunctive model sets a threshold on every dimension judged critical and requires all of them to be cleared: no peak buys back anything. The successive-hurdles model places the tests in series, each one eliminating.

In aviation selection, the arrangement most often encountered combines the first two logics: a few dimensions handled as hard thresholds because they condition the safety of training — multitasking, ability to divide attention, holding up under load — and the rest aggregated in compensatory fashion to rank the remaining candidates. This architecture explains the letter that makes no sense: the candidate was ruled out by a threshold, on a dimension they did not know was critical, even though their overall average placed them very high.

Three ways of aggregating scores and what each of them does with an isolated strength
Model Rule applied What a very high peak contributes What a weak point costs Usual use
Compensatory Weighted sum of the dimensions, compared with a total It raises the total and can carry the application through It is absorbed by the rest if it is not too pronounced Final ranking among candidates already eligible
Conjunctive One threshold per critical dimension, all to be cleared Nothing beyond the threshold of its own dimension It eliminates, whatever the level of the others Dimensions tied to the safety of training
Successive hurdles Tests in series, each one eliminating It does not carry over to the next stage It stops the process at the stage where it appears Selections with a high volume of applications

Aviation selection uses different rules for combining test results. Some systems let you compensate for a weak area with a strong one, while others require you to meet every minimum standard. If you fail one critical test, you might be rejected even with high scores elsewhere.

A candidate for an Airbus A350 position scores 90% on multitasking (a critical dimension) but only 60% on workload management. In a conjunctive system, they would be rejected despite their high multitasking score. In a compensatory system, their high score might compensate for the lower one, but only if workload management isn’t a hard threshold.

Weighted sum or separate thresholds: two verdicts on the same applicationA two-dimensional plane, the first dimension on the horizontal axis, the second on the vertical. A solid oblique line represents the boundary of a compensatory model, where the sum of the two dimensions must exceed a value. Two dashed lines, one vertical and one horizontal, represent two independent thresholds; the shaded area at the top right is the only one that clears both. A point marks a candidate very high on the first dimension and weak on the second: they lie on the right side of the oblique boundary, but outside the shaded area.accepted zonethreshold on the sumthreshold on Bthreshold on Athis candidatedimension Bdimension A
The same application passes or fails depending on the arithmetic chosen. Under a summing rule, a peak buys back a dip. Under separate thresholds it buys back nothing: all that counts is belonging to the upper right quarter.

What the overall score hides: pace, errors, stability

The result of a psychomotor or attention test does not reduce well to a single figure, because it aggregates three independent quantities: response rate, error rate — to be separated into omissions and wrong answers, which say different things about the strategy used — and stability. The same person can hold a high level over the first few minutes of a test and then degrade, something the average hides entirely.

The trade-off between speed and accuracy can be adjusted, and that regulation is precisely what is being measured. A candidate who pushes the pace to the maximum obtains an impressive throughput figure and an error rate that crosses the permitted limit; another, slightly slower, stays under the limit and will be preferred. Neither has “done better”: they have adopted two different regulations, and only one is compatible with the job. The same logic applies to drift at the end of the battery, which says far more about holding up under load than a peak score does.

Your test score isn’t just about how many answers you get right. It also looks at how fast you work, how many mistakes you make, and whether you can keep performing well over time. Just going fast might actually hurt your score if you make too many errors.

On a psychomotor test for a Boeing 787 first officer position, Candidate A completes 120 items with 15 errors (87.5% accuracy) while Candidate B completes 100 items with 5 errors (95% accuracy). Even though Candidate A completed more items, their higher error rate means they might be rejected in favor of Candidate B, who maintained better accuracy.

Response rate and error rate: the usable windowError rate on the vertical axis, response rate on the horizontal. The curve stays low and almost flat as long as the pace is moderate, then rises steeply in the right-hand part. A dashed horizontal line marks the permitted error rate, a dashed vertical line marks the pace at which the curve crosses it. The shaded area under these two lines is the usable domain. A point further to the right, well above the horizontal line, represents a fast candidate whose answers are no longer reliable.usable domainpermitted error ratefast, out of limitserrorsuseful limitpace
Speeding up only pays up to the useful limit. Beyond it, every point of pace gained costs more errors than it brings in answers, and most batteries read the error rate first.

The right reflex. Treat the accuracy instruction as an operating limitation: you do not exceed it to save time. In practice sessions, set your pace by the error rate you observe, not by the number of items completed. If the test announces that errors are penalised, it has already given you the aggregation rule.

The battery does not decide on its own

In a properly built selection process, the psychometric result is not a verdict but a set of hypotheses to be checked. The structured interview takes up the dimensions that left the band and looks for facts: a situation where the workload rose, what was delegated, what was dropped, how a disagreement was handled. Group exercises and simulated situations serve the same function, with behavioural markers observed by several independent assessors and consolidated afterwards.

This is where part of the rejections that seem to contradict the scores is settled. A profile declared cooperative that the interview backs up with no verifiable example becomes a contradiction, not a neutral asset. Conversely, an average score together with a precise, dated operational account, in which the person relates what they misjudged and what they drew from it, weighs more than a high percentile. The documentation in the file acts as a factual counterweight to everything the battery merely suggests. Exchanges between candidates on the FL330 forum in fact describe rejections at this stage far more often than at the testing stage.

A score carries its own margin of error

Every psychometric measurement has a reliability below one, and therefore a standard error of measurement. In concrete terms, a score is not a point but an interval: sitting the test again the next day, the same person would obtain a slightly different result without anything having changed in them. That margin is known to the publisher of the battery and normally appears in its technical manual. It explains the grey zone around the threshold: two candidates whose scores differ by less than that margin cannot be distinguished by the measurement, and an operator who knows this then decides on other elements rather than making the figure say what it does not say.

Two practical consequences follow. The first is the retest policy: the waiting period imposed is not administrative, it targets the practice effect, which inflates scores without improving the aptitude measured at all and contaminates the comparison with the norm. The second is portability: a result obtained with one operator does not hold with another, because neither the battery, nor the norm, nor the target profile, nor the weightings are the same. A rejection travels no better than a success.

There remains the factor nobody writes into a feedback report: the number of vacancies. When the candidates clearing every threshold outnumber the places, the selection shifts from a threshold regime to a ranking regime, and the level of the last candidate admitted then varies from one session to the next without any threshold having moved. This is not an injustice, it is a change in the stopping rule, and it is rarely communicated as such.

Test scores aren’t exact measurements. There’s always some margin of error, meaning your “true” score could be a bit higher or lower than what you actually got. If your score is very close to a passing threshold, the test might not be able to tell for sure whether you passed or failed.

On a cognitive test for an Embraer E-Jet first officer position, the passing threshold is 70. Candidate X scores 69, Candidate Y scores 71. If the test’s margin of error is ±2 points, Candidate X’s “true” score could actually be 71 (passing) and Candidate Y’s could be 69 (failing). In this case, the selection committee would look at other factors to make their decision.

Three results and their measurement margin against a thresholdA horizontal score scale and a dashed vertical line marking the threshold. Three horizontal bars each represent a candidate: a central point and the interval within which the true score lies given the reliability of the test. The first bar is entirely to the left of the threshold, the third entirely to the right, the second straddles it: its central point is on the right side, but its interval crosses the threshold. For this third case, the decision rests on the measurement margin as much as on performance.thresholdbelowstraddlingabovemeasurement marginscore
A threshold is a line, a score is an interval. The middle case explains decisions that appear arbitrary: the value displayed clears the bar, the measurement margin does not guarantee that it does. Many operators then decide on other elements of the application.

What to ask before the next session

A few questions asked at the right moment are worth more than an appeal. Before registering, ask whether an individual feedback report is provided, what retest interval applies, in what language and on what equipment the tests are run. After a rejection, ask whether the decision came from one dimension in particular or from the overall aggregation: the answer is not always given, but the question is legitimate, and a decision resulting from entirely automated processing opens, in Europe, under data protection law, a right to information and the possibility of requesting human intervention.

Two men in suits in an office, one using a computer while the other observes.
Performance tests don't guarantee job offers; other factors matter.

For preparation, the dividing line is clear. Becoming familiar with the formats — instructions, interfaces, time constraints — removes needless noise. Running hundreds of repetitions of the same test moves the score without moving the aptitude, and it backfires if the next test comes from another publisher. What really can be worked on lies upstream: the regulation between speed and accuracy, sustaining attention over a long period, and above all interview material, that is, real situations one is able to recount along with what one got wrong in them. The programmes offered on formations.fl330.com cover that part of the preparation, the part that is not played out in front of a screen.

A grey area remains and needs naming: nothing obliges an operator to publish its target profile, its weightings or its thresholds, and the confidentiality of the scoring keys is even a condition of their validity. The candidate is therefore working on a system they will never see in full. This opacity is also why relentless efforts to decode the test are a poor investment, whereas the quality of the application file and the precision of the professional account remain entirely under control.

Each operator’s selection system, the technical manual of the battery used and the part of the operations manual devoted to training take precedence over everything described here: this article sets out general mechanisms, not the procedure of any given employer.

Questions fréquentes

Does a high percentile announced verbally guarantee eligibility?

No, because a percentile only makes sense in relation to the reference group used for the norming. The same raw score can be a tail of the distribution against a general population and a median result against the pool of candidates who have already sat that selection. As long as the norm is not named, the figure allows no conclusion about the application’s real standing.

Can you be ruled out for a score that is too high on one dimension?

Yes, on the behavioural dimensions, whose expected profile is a range bounded above as well as below. Assertiveness or concern for detail far above the band reads as a risk for crew work: the first can silence the other pilot’s contribution, the second can lock the crew onto a minor anomaly. On the purely cognitive dimensions, by contrast, a high level is never penalising in itself.

Why did an excellent overall result not make up for a single poor test?

Because the selection process probably does not apply a compensatory rule on that dimension. Capacities judged critical for the safety of training, such as divided attention or holding up under load, are often handled by a threshold to be cleared independently of the rest. Under this conjunctive logic, no peak obtained elsewhere buys back the threshold that was missed, and the overall average only serves to rank candidates who are already eligible.

Does practising on online test simulators improve your chances?

Familiarisation with the formats, the instructions and the time constraints removes needless noise and is justified. Intensive repetition of the same exercise, on the other hand, moves the score without moving the aptitude measured and distorts the comparison with the norm, since the reference group did not have the benefit of the same practice. The benefit is lost in any case as soon as the next test comes from another publisher, whereas the return remains high on the regulation between speed and accuracy and on preparing interview material.

Is a result obtained with one operator usable with another?

As a general rule, no. The battery, the norming, the target profile for the job and the weightings differ from one employer to another, and a selection system is only valid for the job it was built for. A rejection travels no better than a success, which explains why candidates turned down on one side are taken on the other without anything having changed in their aptitudes.

Why is a waiting period imposed before retaking the tests?

The waiting period targets the practice effect: taken again after a short interval, the same test gives a higher score simply through familiarisation, with no improvement in aptitude. That artificial rise would make the comparison with the norm wrong and the decision less reliable. The waiting period therefore protects the validity of the measurement, not the organisation of recruitment.

Commentaires

Leave a Reply

Découvrez l’écosystème FL330

FL330, c’est toute l’aviation professionnelle — explorez nos autres espaces.