Free Guide · Risk Management
Bias Testing and Fairness Guide
Fairness testing is a control. This guide gives security, risk, and compliance teams a repeatable process to evaluate AI systems for bias, document results, and demonstrate governance to auditors.
A practical checklist
Define the fairness objective
- •What decision is the model influencing (hiring, lending, support triage, fraud flags)?
- •What harm are you trying to prevent (denials, delays, false accusations, differential quality of service)?
- •What populations are in scope (customers, employees, applicants)?
Choose metrics
- •Group parity metrics (selection rate, false positive/negative rates)
- •Calibration/score reliability across groups
- •Distribution shift monitoring
- •For GenAI: toxicity, stereotyping, and refusal consistency
Design test datasets
- •Collect representative samples for each group (or use controlled synthetic data where appropriate)
- •Define ground truth labels and reviewer guidance
- •Log dataset provenance and consent/collection constraints
Run evaluations
- •Evaluate metrics overall and per group
- •Test on edge cases and adversarial prompts (for GenAI)
- •Repeat across model versions and configuration changes
Document results
- •Record metrics, thresholds, and pass/fail criteria
- •Attach evidence (dashboards, notebooks, test logs)
- •Summarise limitations and known failure modes
Remediate and retest
- •Apply mitigations (data balance, post-processing, policy controls, guardrails)
- •Rerun the full evaluation suite
- •Implement monitoring to detect regression
What to include in a fairness report
•Model/version and configuration
•Intended use + out-of-scope use
•Groups evaluated and data sources
•Metrics and thresholds
•Results and identified gaps
•Mitigations applied
•Residual risk acceptance
•Monitoring plan and alert thresholds
Complete AI governance library
Get all 39 templates in one ZIP
Policies, registers, checklists and rollout plans, ready to customise.
- 46 files (DOCX, XLSX, PPTX)
- 2.0 MB total
- Updated October 2026
- NIST AI RMF · ISO 42001 · EU AI Act aligned
Work email only. Aona AI provides a SOC 2 Type 2 report for security review.
Get started
Want continuous governance evidence?
A guide helps you run point-in-time tests. Aona AI gives you continuous visibility into AI usage, policy enforcement, and audit-ready evidence.