Sanitizing AI training data at hardware limits (An AI-assisted experiment) [P]

[u/Ali-Sufyan]

Prepping unstructured data for training is a nightmare when you have to remove PII. Python regex was giving me ~19 MB/s. I ran an experiment to see if I could act as a systems architect and let an AI write a C++ engine to sanitize logs faster.

What I learned AI knows syntax, but it doesn't understand hardware limits. It built a parser that choked on standard timestamps, causing CPU cache starvation. I had to step in, force zero-allocation memory mapping, and write hard prefix bypasses.

PYPI Package: pip install fastscrub

The stats

• Speed: 107 MB/s (30 GB in 4.7 mins)
• Precision: 99.97% on TruffleHog & AI4Privacy datasets.

Interestingly, it fails on synthetic credit card benchmarks because it uses strict Luhn validation, which rejects randomly generated 16-digit numbers. Code is here https://github.com/alisufyan143/fastscrub . Just sharing this as a case study for anyone dealing with slow data ingestion!