Training Hub › Data Types & Ingestion

PII Detection & Redaction

Data Engineering & Processing beginner Building with AI 12 min read

Overview

Personally Identifiable Information (PII) — names, email addresses, phone numbers, social security numbers, credit card numbers, and similar data — poses a significant risk when it leaks into AI systems. If PII gets embedded into vector stores or passed to third-party LLM APIs, you face regulatory fines (GDPR penalties can exceed €20 million or 4% of global revenue), reputational damage, and genuine harm to the individuals whose data is exposed.

PII detection and redaction is the practice of automatically identifying sensitive information in text and either removing it, masking it with placeholder tokens, or replacing it with synthetic equivalents before the data enters your AI pipeline. This is not optional for production systems — it’s a fundamental requirement of responsible AI engineering.

The challenge is balancing thoroughness with accuracy. Overly aggressive detection creates false positives that strip useful context from your data. Under-detection lets sensitive information slip through. Modern approaches use a hybrid architecture combining fast regex pattern matching with contextual NER (Named Entity Recognition) models to achieve both speed and accuracy, with hybrid systems reaching F1 scores above 91% on real-world datasets.

How It Works

What Counts as PII?

PII definitions vary by regulation, but generally include:

Key Concept: The Re-identification Risk

Research by Latanya Sweeney showed that 87% of the U.S. population can be uniquely identified by the combination of zip code, date of birth, and gender alone. PII detection must account for quasi-identifiers, not just obvious fields like SSN or email.

Detection Technique 1: Regex Pattern Matching

Regular expressions are the fastest first-pass filter for structured PII. Patterns for credit cards, SSNs, phone numbers, emails, and IP addresses are well-defined and can be matched in microseconds per document.

# Common PII regex patterns
import re

PII_PATTERNS = {
    "email": re.compile(
        r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b"
    ),
    "ssn": re.compile(
        r"\b\d{3}-\d{2}-\d{4}\b"
    ),
    "credit_card": re.compile(
        r"\b(?:\d{4}[- ]?){3}\d{4}\b"
    ),
    "phone_us": re.compile(
        r"\b(?:\+1[-.\s]?)?\(?\d{3}\)?[-.\s]?\d{3}[-.\s]?\d{4}\b"
    ),
    "ip_address": re.compile(
        r"\b(?:\d{1,3}\.){3}\d{1,3}\b"
    ),
}

def detect_pii_regex(text: str) -> list[dict]:
    findings = []
    for pii_type, pattern in PII_PATTERNS.items():
        for match in pattern.finditer(text):
            findings.append({
                "type": pii_type,
                "value": match.group(),
                "start": match.start(),
                "end": match.end(),
            })
    return findings

Regex is deterministic and fast, but it has limitations: it cannot detect names, addresses in free-form text, or context-dependent PII. The string “John Smith visited 123 Main Street” contains two pieces of PII that no regex can reliably catch without a dictionary or model.

Detection Technique 2: Named Entity Recognition (NER)

NER models — typically transformer-based architectures like BERT fine-tuned on PII-annotated data — provide contextual detection. They understand that “Apple” in “Apple released a new phone” is an organization, not a person, while “Apple” in “Apple Johnson called yesterday” is likely a name.

Key Concept: The Hybrid Architecture

Production PII detection uses a two-tier pipeline: regex runs first as a fast-pass filter for well-structured PII (SSNs, credit cards, emails), then an NER model processes remaining text for contextual entities (names, addresses, medical terms). This achieves both speed and accuracy — regex handles ~60% of detections in microseconds, leaving only the ambiguous cases for the more expensive model.

Popular NER models for PII include:

Detection Technique 3: Checksum Validation

For structured identifiers, pattern matching alone isn’t enough — you need validation. Credit card numbers follow the Luhn algorithm; SSNs have area-group-serial constraints; IBANs have modular check digits. Validating against these checksums dramatically reduces false positives.

# Luhn algorithm for credit card validation
def luhn_check(number: str) -> bool:
    digits = [int(d) for d in number if d.isdigit()]
    odd_digits = digits[-1::-2]
    even_digits = digits[-2::-2]
    total = sum(odd_digits)
    for d in even_digits:
        total += sum(divmod(d * 2, 10))
    return total % 10 == 0

Redaction Strategies

Once PII is detected, you have several options for handling it:

Microsoft Presidio: The Industry Standard

Microsoft’s open-source Presidio framework is the most widely adopted PII detection and anonymization toolkit. It has two core components:

# Using Microsoft Presidio for PII detection and redaction
from presidio_analyzer import AnalyzerEngine
from presidio_anonymizer import AnonymizerEngine

analyzer = AnalyzerEngine()
anonymizer = AnonymizerEngine()

text = "John Smith's SSN is 123-45-6789 and his email is john@example.com"

# Detect PII entities
results = analyzer.analyze(
    text=text,
    entities=["PERSON", "US_SSN", "EMAIL_ADDRESS"],
    language="en"
)

# Redact detected entities
anonymized = anonymizer.anonymize(text=text, analyzer_results=results)
print(anonymized.text)
# Output: "<PERSON>'s SSN is <US_SSN> and his email is <EMAIL_ADDRESS>"

Practical Applications

RAG Pipeline Pre-processing

Before documents enter your embedding pipeline, run them through PII detection. Redact or mask sensitive information so that it never gets encoded into vector representations. This prevents the scenario where a semantic search for “project manager contact” retrieves an actual person’s phone number and email from the vector store and presents it to any user who asks.

LLM API Gateway Protection

When sending prompts to third-party LLM APIs (OpenAI, Anthropic, Google), a PII detection layer at the API gateway intercepts outbound requests and redacts sensitive data before it leaves your infrastructure. The added latency is typically negligible (10–50ms) compared to LLM inference time (hundreds of ms to seconds), but prevents compliance violations.

Observability and Logging

LLM tracing systems (like OpenTelemetry-based observability pipelines) log prompts and completions for debugging. Without PII redaction in the logging pipeline, sensitive user data ends up in log aggregation systems accessible to engineering teams — a compliance nightmare. A PII redaction middleware between the LLM and the trace collector prevents this leakage.

Training Data Curation

If you’re fine-tuning models on proprietary data, PII in the training set can be memorized by the model and regurgitated during inference. Pre-training PII scrubbing ensures that the model never learns to associate specific individuals with specific attributes.

Key Takeaways

Further Reading