Benchmarks
Methodology
Benchmark evaluation methodology for FastPII detection accuracy.
Methodology
Test corpus
The benchmark corpus consists of reference files covering all detector types across supported regions.
| Region | Files | Description |
|---|---|---|
| CZ | Multiple | Czech identifiers in natural text |
| PL | Multiple | Polish identifiers in natural text |
| DE | Multiple | German identifiers in natural text |
| FR | Multiple | French identifiers in natural text |
Each file contains ground truth annotations with:
- Expected PII type
- Expected value
- Expected character positions (start, end)
- Difficulty level (easy, medium, hard)
Metrics
FastPII uses the following metrics:
| Metric | Formula | Description |
|---|---|---|
| Precision | TP / (TP + FP) | How many detected findings are correct |
| Recall | TP / (TP + FN) | How many actual findings were detected |
| F1 | 2 * P * R / (P + R) | Harmonic mean of precision and recall |
| F2 | 5 * P * R / (4P + R) | Weighted toward recall |
| FP/1000w | FP / words * 1000 | False positives per 1000 words |
IoU matching
Findings are matched against ground truth using Intersection over Union (IoU):
IoU = overlap_area / union_areaA finding is considered a true positive if IoU >= threshold (default: 0.5).
Normalized comparison
All comparisons use normalized values:
- Whitespace and case are normalized
- Formatting characters are stripped
- Checksum-validated identifiers have higher accuracy
Ground truth validation
Each ground truth file is validated against the checksum algorithms before benchmarking to ensure correctness.