Short answer
Audit predictive people analytics from the decision backwards. Define the outcome, unit and time horizon; check that every feature existed before the prediction date; compare a simple baseline; test on unseen later data; examine false positives, false negatives and calibration; review errors across relevant groups; and prevent a probability from becoming an unsupported judgement about a person.
Ask for an evidence pack, not a headline metric
“Accurate model” has little meaning without the outcome, population, date, threshold and test method. A buyer should be able to reconstruct what the model estimated and how its performance was assessed.
Request this evidence pack:
| Record | Question it must answer |
|---|---|
| Intended-use statement | Which decision may the output inform, and which uses are excluded? |
| Outcome contract | What exactly counts as the outcome, by what date and for which population? |
| Data dictionary | Which fields and derived variables enter the model? |
| Feature-time record | Was each value genuinely available at the prediction date? |
| Dataset split | Which periods and people were used for training, validation and final testing? |
| Baseline | What simple rule or rate must the model improve on? |
| Metric table | Base rate, confusion matrix, precision, recall, calibration and uncertainty |
| Group review | Where do errors or missing data differ under a justified lawful analysis? |
| Decision test | What happens to people and work when the output is used? |
| Change record | Which data, model, threshold or context change requires retesting? |
| Stop rule | Which failure pauses use, who decides and what route replaces it? |
This is narrower than a general ethical AI in HR method. That guide covers instructions, source handling, human review and incidents across AI uses. This article tests the statistical evidence behind a predictive model.
Define the outcome contract
A model cannot be assessed against “talent risk.” Write a testable outcome such as “voluntary departure recorded within six months of the forecast date.” Then settle details that could change the label:
- whether internal transfers count;
- how retirement, redundancy and contract end are treated;
- whether a withdrawn notice counts;
- which employees are eligible at the forecast date;
- when the observation window closes;
- how delayed or corrected records are handled;
- which decisions the output is prohibited from informing.
Changing the label changes the model. Preserve the label version used in every evaluation.
Check feature timing and leakage
Feature leakage occurs when training data contains information that would not have been available at the moment of use. A final payroll status, exit interview code, replacement requisition or retrospective manager entry can reveal an outcome that the supposed forecast was meant to anticipate.
For every feature, record:
- source system and owner;
- event date and entry date;
- value available at the forecast cut-off;
- missing-value rule;
- later corrections;
- possible proxies for protected or sensitive characteristics;
- whether the field changes because of an earlier model output.
The current Google Machine Learning guidance on dividing datasets recommends separate training, validation and test sets, with a test set that represents real use and contains no training duplicates. For a time-dependent workforce use, also test on later periods so future records cannot leak backwards.
Compare with the base rate
Suppose a fictional validation set contains 1,000 employees, of whom 100 later have the defined outcome. A model flags 250 people. Seventy are true positives, 180 are false positives, 30 are false negatives and 720 are true negatives.
| Metric | Fictional result | Interpretation |
|---|---|---|
| Base rate | 10% | 100 of 1,000 have the defined outcome |
| Recall | 70% | 70 of the 100 outcomes were flagged |
| Precision | 28% | 70 of the 250 flags matched the outcome |
| Accuracy | 79% | 790 of all 1,000 labels were correct |
A rule that predicts “no outcome” for everyone would have 90% accuracy in this fictional set. It would also identify none of the outcomes. The model's 79% accuracy therefore does not demonstrate value, even though it catches more true outcomes.
Google's current guide to accuracy, precision and recall explains why accuracy can mislead on imbalanced data and why metric choice depends on the task and costs of each error. In employment, the practical and human consequences of a false label matter as well as the arithmetic.
Do not use this example to select a threshold. It demonstrates the questions a buyer must ask. A proposed threshold needs its own decision, error costs, evidence and review.
Test probability calibration
A ranking metric does not show whether a stated probability is trustworthy. Calibration asks whether outcomes occur at about the stated rate among comparable predictions. If 100 independent cases are assigned probabilities near 0.7 and only 40 meet the outcome under the agreed definition and period, that group of probabilities is overconfident in this fictional example.
The current scikit-learn probability-calibration guide describes calibration curves that compare mean predicted probability with the observed fraction in each bin. It also notes that calibration should use data independent from the model's training data.
Ask for:
- a calibration plot and counts in each band;
- results on unseen data from the intended population;
- uncertainty where bands contain few observations;
- performance before and after any recalibration;
- a clear distinction between ranking ability and probability accuracy.
A probability is still a model output. It is not evidence of an individual's intention and does not explain why an outcome occurs.
Review errors in context
An overall result can hide poor performance in a role, location, work pattern or other relevant group. Decide with legal and subject-matter owners which comparisons are appropriate, which data can lawfully be used and what sample sizes support interpretation.
The NIST AI RMF trustworthiness guidance says accuracy measures should use clearly defined, realistic test sets and consider false-positive and false-negative rates, generalisation and the fact that some failures cause greater harm. NIST's framework is voluntary and cross-sector.
The UK Information Commissioner's Office guidance on fairness, bias and discrimination warns that removing protected attributes does not necessarily remove correlated proxies and that statistical fairness metrics do not establish legal or substantive fairness by themselves.
For US employment uses, the EEOC's 2024 explainer on its role in AI says the federal anti-discrimination laws it enforces can apply when AI is used in recruiting, assessing worker performance, and making decisions about pay, promotion or termination. The applicable review depends on the use and jurisdiction. A vendor metric is not a legal conclusion.
Test the decision made around the model
Even a statistically valid estimate may be unsuitable for the proposed workflow. Run a decision simulation with fictional records:
- What does the reviewer see besides the output?
- Can they inspect relevant source and uncertainty?
- Do they have time, authority and expertise to disagree?
- Is the employee told about a consequential use through the applicable process?
- Can an affected person correct data or challenge an action?
- Does the output narrow access to work, pay, promotion, development or support?
- Could the same purpose be achieved with an aggregate forecast or universal invitation?
- What happens when the model is unavailable or paused?
“A human reviews it” is incomplete if the reviewer sees only a probability and is expected to accept it.
Track change and set stop rules
Retest when the outcome definition, data source, model, threshold, population, policy, labour market or operating process changes materially. Keep a dated record of actual performance and affected-user reports.
Pause the model when:
- feature timing or outcome labels cannot be reconstructed;
- the final test set has contaminated or duplicate records;
- a material group has too little evidence for the claimed use;
- false-positive or false-negative costs exceed the approved boundary;
- calibration falls outside the agreed tolerance;
- users convert the output into an unauthorised employment decision;
- people cannot use the correction or challenge route;
- a material change has occurred without retesting.
Name the person authorised to pause use, the alternative process and the evidence required before restart.
A buyer audit decision
A fictional UK and US business is offered a model said to identify voluntary-departure risk. The vendor supplies a strong ranking metric but no outcome contract, chronological test, calibration plot or false-positive count. HR also learns that a replacement requisition can enter the dataset before an exit is formally recorded, creating possible leakage.
The buyer does not run the model on employees. It asks the vendor to reproduce the evaluation with a fixed outcome definition, only data available at the forecast cut-off, later unseen periods, a simple base-rate comparison, confusion matrices, calibration bands and relevant group results. It also requests a fictional-record workflow test and a route that does not expose an individual label to a manager.
Until that evidence exists, the buyer records the capability as unverified. It uses an aggregate capacity forecast for planning and offers transparent work conversations through ordinary processes. This preserves the business decision without turning uncertainty into a claim about a person.
Where Lontra fits
If a defined work question needs recent employee context, explore Lontra's employee conversations and manager briefs. Managers receive a brief rather than raw employee conversations, and people decide what to do.
Lontra does not provide this predictive model audit, calculate employee risk, predict departures or make automated employment decisions. Conversation material should not be repurposed as an individual label or as proof of prevalence or cause.
Frequently asked questions
How do you audit a predictive people analytics model?
Define the outcome, unit, horizon and decision first. Then inspect labels and feature timing, compare a simple baseline, test on unseen later data, review false positives and false negatives, assess probability calibration and group errors, and set change and stop rules.
Is model accuracy enough for predictive people analytics?
No. Accuracy can look strong when the outcome is uncommon. Review the base rate, confusion matrix, precision, recall, probability calibration, error costs and performance in relevant groups and operating conditions.
Can a probability explain why an employee outcome happens?
No. A predictive association is not a causal explanation. Test possible mechanisms with suitable quantitative and qualitative evidence, and do not convert a probability into a statement about a person's intention or character.