10+ Years In Business | 4 Continents |
16+ Countries | 32+ Locations

Contact Us

AI and Machine Learning for Enhanced Document and Image Classification with PII Detection

Project Overview

This case study demonstrates the strategic use of various AI/ML models to create a comprehensive solution that not only meets classification and PII extraction needs but also ensures data privacy and regulatory compliance, streamlining the entire document and image handling process.

Project sought to automate the classification, organization, and information extraction of large volumes of documents and images. Our client’s primary goals included efficient categorization of various types of documents, precise object detection, and secure extraction of Personally Identifiable Information (PII). Given the sensitivity of the data and volume of unstructured information, the project demanded high accuracy, custom fine-tuning, and advanced AI tools.

Industry

Cybersecurity / Data Security

Tech Stack

  • YOLO (You Only Look Once)
  • NER (Named Entity Recognition)
  • PaddleOCR
  • Donut (Document Understanding Transformer)
  • AWS Textract
  • BERT
  • PHI-3 Vision
  • Azure Document AI

Team
Composition

  • 1 Project Manager
  • 1 UX/UI
  • 6 Backend Dev
  • 7 Frontend Dev
  • 3 Data Sciences
  • 4 QA
  • 2 Cloud Infra & DevOps
  • 2 Engineers

Client Background

Industry leader specializes in Data Breach Response, PII & PHI Detection, Data Security, Sensitive Information Detection, Privacy, Data Subject Access Requests, Incident Response, Data Protection, GDPR, CCPA, Incident Response, and Cybersecurity.

The Challenge

  1. Data Sensitivity: Securing data was challenging. Models were deployed in a secure cloud environment, with access controls and encryption protocols.
  2. Accuracy Demands: Fine-tuning each model was labor-intensive but was addressed by leveraging synthetic datasets and manual labeling.
  3. Handling Diverse Document Formats: Complex layouts posed challenges for document parsing. Using a combination of Donut, Textract, and PaddleOCR, the system achieved robust adaptability across formats.

The Solution:

  1. Data Ingestion & Preprocessing
  • Data Collection: Diverse datasets of medical records, forms, images, and handwritten notes were ingested.
  • Preprocessing: Images were resized and enhanced for OCR clarity. Documents were categorized by type for more accurate model fine-tuning.

2. Image and Document Classification

  • Donut (Document Understanding Transformer): Using Donut, documents were categorized by type (e.g., insurance forms, prescriptions, invoices,bank statements, government ids, tax forms). The model was fine-tuned on a custom dataset for improved accuracy.
  • BERT: BERT was used to analyze document content to further classify documents by specific topics based on detected text patterns.

3. Object Detection with YOLO

  • Custom Training: YOLO was trained on images to recognize objects like government ids, forms
  • Application in Document Workflows: YOLO helped identify relevant document sections, such as patient information headers, and detected medical icons or stamps that signified importance.

4. OCR and Text Extraction

  • PaddleOCR and AWS Textract: For clear text recognition, PaddleOCR handled document images with text, while AWS Textract extracted data from forms and tables with high accuracy. Textract was also optimized for different languages and layouts, critical for documents from international sources.
  • PHI-3 Vision: This was employed specifically for PII detection, and to find relation between extracted PII

5. PII Detection and Redaction

  • NER and BERT for PII Identification: A customized Named Entity Recognition (NER) model, integrated with BERT, helped identify PII entities (names, addresses, phone numbers) within text blocks. Fine-tuning was done on a dataset labeled with specific PII entities to improve detection accuracy.
  • Azure Document AI: Document AI identified and extracted entities from complex document layouts, helping to process and redact PII from difficult forms, hand-written notes, and tables.
  • Custom Logic for PII Redaction: PII entities were flagged, and an automatic redaction module was applied to all flagged areas in compliance with data protection requirements.
Download PDF
Contact Us!
Linkedin-in X-twitter Instagram

©2025 CANWILL ATHENAWORKS HOLDINGS LLC