I Finally Achieved Automatic ID Card and Face Capture on Web Pages, Face Similarity Comparison Between Images and Videos, and...

🎯 Project Overview

Context: Building a production-grade KYC (Know Your Customer) verification system from scratch
Timeline: 3 months intensive development
Team Size: Solo developer (with backend infrastructure support)
Business Impact: Critical for company compliance and user onboarding

I was tasked with leading the development of our company's core KYC system. This wasn't just a technical challenge - it was a business-critical project that would determine whether our company could scale user onboarding while maintaining regulatory compliance. The system needed to handle thousands of verification attempts daily with 99.9% accuracy.

📋 Technical Requirements & Constraints

Business Requirements

• Accuracy: >99% face recognition accuracy
• Speed: Complete verification within 2 minutes
• Availability: 99.9% uptime with no data loss
• Scalability: Handle 10,000+ concurrent verifications
• Compliance: GDPR and local data protection regulations

Technical Constraints

• Environment: Mixed GPU/CPU infrastructure
• Languages: Korean ID cards with English support
• Platforms: Web-based with mobile optimization
• Storage: Efficient handling of large video files
• Real-time: WebSocket-based progress updates

🏗️ System Architecture Design

Technology Stack Decision Process

Initially, I analyzed and compared several face recognition libraries based on specific criteria:

Face Recognition Engine Comparison:

interface FaceEngine { name: string; accuracy: number; speed: 'fast' | 'medium' | 'slow'; license: 'free' | 'commercial'; gpuSupport: boolean; stability: number; // 1-10 scale } const engines: FaceEngine[] = [ { name: 'InsightFace', accuracy: 99.8, speed: 'fast', license: 'free', gpuSupport: true, stability: 7 }, { name: 'OpenCV YuNet/sFace', accuracy: 97.2, speed: 'medium', license: 'free', gpuSupport: true, stability: 9 } ]; 

Selection Matrix:
| Criteria | InsightFace | OpenCV | Winner |
|----------|------------|--------|--------|
| Accuracy | ✅ 99.8% | ❌ 97.2% | InsightFace |
| Stability | ❌ Medium | ✅ High | OpenCV |
| License | ✅ Free | ✅ Free | Tie |
| GPU Support | ✅ Yes | ✅ Yes | Tie |

Final Decision: Hybrid approach combining both engines

API Design Pattern

The system follows a RESTful API design with WebSocket support for real-time updates:

// Core API Endpoints interface KYCApiSpec { // ID Card Processing 'POST /api/v1/id-capture': CaptureRequest; 'GET /api/v1/id-capture/{sessionId}': CaptureStatus; // Face Recognition 'POST /api/v1/face-video': VideoUploadRequest; 'POST /api/v1/face-similarity': SimilarityRequest; 'GET /api/v1/face-similarity/{comparisonId}': SimilarityResult; // Real-time Updates 'WS /ws/kyc/{sessionId}': WebSocketUpdates; } // Response Schema Standards interface APIResponse<T> { success: boolean; data?: T; error?: { code: string; message: string; details?: any; }; timestamp: string; requestId: string; } 

Overall System Architecture

Frontend (React 19 + TypeScript) ↓ WebSocket Backend (FastAPI + SQLAlchemy) ↓ Async Tasks Celery Workers ↓ Database MariaDB + Redis 

🔥 Phase 1: Face Recognition Dual Engine Implementation

The Hybrid Strategy: Why Two Engines Are Better Than One

I discovered that no single face recognition engine could handle all real-world scenarios. InsightFace offered incredible accuracy (99.8%) but failed in poor lighting, while OpenCV was rock-solid but slightly less accurate.

The Solution: A dual-engine system that automatically switches between engines based on conditions:

Key Innovations:

• Smart Hardware Detection: Automatic GPU/CPU adaptation
• Memory Management: Singleton pattern prevents GPU memory leaks
• Fallback Logic: Seamless engine switching based on confidence scores

Impact: Success rate jumped from 92% to 99.9% by combining both engines' strengths.

🎥 Phase 2: Video-Image Similarity Comparison

From 3 Minutes to 6 Seconds: The Video Processing Revolution

My initial approach processed all 900 frames of a 30-second video - taking over 3 minutes and often crashing servers. The breakthrough was realizing most frames were redundant.

Smart Sampling Strategy:

Key Innovations:

• Frame Sampling: Reduced from 900 to 12 frames (98.7% reduction)
• Quality Filtering: Only frames >80% quality used
• Cosine Similarity: 512-dimensional embeddings for accurate comparison

Result: Processing time dropped from 180+ seconds to 6 seconds while actually improving accuracy.

📸 Phase 3: Automatic ID Card Capture

The Korean OCR Challenge: Teaching Computers to Read Hangul

Most OCR systems fail with Korean characters. After testing Tesseract, EasyOCR, and cloud services, I discovered PaddleOCR which had surprisingly good Korean support, but required extensive fine-tuning.

Automatic Quality Assessment Pipeline:

Four Quality Metrics:

• Sharpness: Laplacian variance for blur detection
• Lighting: Even illumination without glare
• Angle: Perspective distortion detection
• Completeness: All four corners visible

Result: User completion rate jumped from 60% to 95% by eliminating manual capture timing.

🗄️ Phase 4: Database & Asynchronous Processing

The Scalability Architecture: Handling Thousands at Once

Traditional synchronous processing would make users wait 5-10 seconds - unacceptable for KYC. The solution was a complete asynchronous revolution.

Async Processing Pipeline:

Key Innovations:

• Hybrid Storage: Database metadata + filesystem for large files
• Complex Relationships: Many-to-many image/video similarity mappings
• Distributed Tasks: Celery + Redis for reliable processing
• Real-time Updates: WebSocket connections for live progress

Impact: System handles 100x more concurrent users with zero perceived delay.

🔄 Phase 5: Real-time User Experience

Zero-Wait Processing: The WebSocket Revolution

Users need instant feedback, not spinning loaders. The challenge was maintaining real-time connections for thousands of simultaneous KYC sessions.

Real-time Communication Flow:

Key Frontend Innovations:

• Progress Visualization: Multi-stage progress bars with specific feedback
• Smart Error Handling: User-friendly guidance instead of cryptic errors
• Mobile Optimization: Touch-friendly interface with camera quality detection
• State Recovery: Automatic recovery after page refreshes

Connection Management: Heartbeat mechanisms prevent memory leaks, automatic reconnection handles network drops, session persistence maintains processing state.

Result: Users never feel like they're waiting - they see exactly what's happening at every step.

🐛 Major Debugging Process

Problem 1: The Ghost in the GPU - Memory Leaks

The Crisis: After a week of successful testing in production, the system suddenly started crashing every 4-6 hours. The pattern was always the same - gradual memory increase followed by a complete system freeze. At first, I thought it was a regular memory leak, but monitoring showed RAM usage was stable. The culprit was GPU memory.

Step-by-Step Problem Resolution:

1. Problem Identification

   • Symptom: System crashes every 4-6 hours
   • Initial diagnosis: Memory leak
   • Tools: nvidia-smi, system monitoring

2. Hypothesis Testing

   • Theory 1: Regular RAM leak → ❌ RAM usage stable
   • Theory 2: GPU memory leak → ✅ GPU memory steadily increasing
   • Evidence: Each face recognition call added 50-100MB GPU memory

3. Root Cause Analysis

   • Location: InsightFace model initialization
   • Issue: GPU contexts not released after inference
   • Impact: Cumulative memory allocation

4. Solution Implementation
   

 # Memory management workflow  def process_with_memory_cleanup(): try: # Face recognition operation  result = insightface_app.process(frame) return result finally: # Critical: Explicit GPU cleanup  if torch.cuda.is_available(): torch.cuda.empty_cache() torch.cuda.synchronize() 

1. Prevention Measures
   • Memory monitoring with automatic thresholds
   • Service restart automation
   • Regular memory usage reporting

The Learning: GPU memory management requires explicit cleanup. Python's garbage collector doesn't automatically free GPU resources, leading to cumulative memory leaks that can crash production systems.

Problem 2: The Time Traveling Video Frames

The Bizarre Bug: During testing, I noticed something impossible - sometimes the similarity calculations would show results that didn't make sense, like comparing a face from the beginning of a video with one from the end, but the timestamps would suggest they were consecutive frames.

Step-by-Step Debugging:

1. Anomaly Detection

   • Symptom: Similarity scores didn't match expected frame progression
   • Evidence: Frame timestamps didn't align with calculated similarities
   • Impact: Random accuracy drops

2. Root Cause Investigation
   

 # Problem: OpenCV's internal buffering  cap.set(cv2.CAP_PROP_POS_FRAMES, target_frame) # Requested frame  ret, frame = cap.read() # Got buffered frame instead! 

1. Solution Implementation

 # Frame precision control  cap.set(cv2.CAP_PROP_BUFFERSIZE, 1) # Minimize buffer  cap.set(cv2.CAP_PROP_POS_FRAMES, target_frame) ret, frame = cap.read() # Validation step  actual_frame = int(cap.get(cv2.CAP_PROP_POS_FRAMES)) if actual_frame != target_frame: # Handle frame mismatch 

Problem 3: The Concurrent Catastrophe

The Meltdown Scenario: During load testing with just 10 concurrent users, the system started producing completely wrong results. Users would get similarity scores that belonged to completely different people. This was a critical security and privacy issue that could have had serious consequences.

Crisis Management Steps:

1. Incident Response (Minutes)

   • Immediate system shutdown
   • Alert security team
   • Preserve logs for forensics

2. Root Cause Analysis (Hours)
   

 # Problem: Shared singleton instance  class FaceRecognitionService: _instance = None # Shared across all requests! ❌  # Solution: Service pooling  class FaceRecognitionPool: def __init__(self, pool_size=5): self.pool = [FaceRecognitionService() for _ in range(pool_size)] self.available = Queue() def get_service(self): return self.available.get() def return_service(self, service): self.available.put(service) 

1. Security Validation

   • Multi-threaded testing with 100+ concurrent requests
   • Result verification: No cross-contamination
   • Performance testing: Maintained throughput

2. Production Safeguards

   • Comprehensive logging for all face recognition operations
   • Request correlation tracking
   • Automated anomaly detection

Learning: Thread safety is not optional for biometric systems. Always design for concurrency from day one, especially when dealing with sensitive user data.

📊 Performance Optimization Results

Processing Speed Improvements

The biggest win was video processing - reducing a 3-minute ordeal to just 6 seconds completely changed the user experience. Users went from abandoning the process to completing it successfully.

Face Recognition Engine Performance

The hybrid approach gave us the best of both worlds - InsightFace's industry-leading accuracy when conditions are good, and OpenCV's rock-solid reliability as a safety net. This increased our overall success rate from about 92% to 99.9%.

🎯 Final System Architecture

💡 Key Learning Points

Technical Growth

1. Advanced Computer Vision: Practical experience with diverse CV libraries like InsightFace, OpenCV, and PaddleOCR
2. Performance Optimization: GPU memory management, asynchronous processing, caching strategies
3. System Architecture: Experience designing microservices and event-driven architectures
4. Database Design: Optimization for large-scale media data storage and retrieval

Project Management

1. Technology Selection Process: Experience with accuracy vs performance vs stability trade-offs
2. Incremental Development: Methods for implementing complex systems step by step
3. Problem-solving Skills: Experience resolving memory leaks, concurrency, and performance issues
4. Documentation: Understanding the importance of systematic recording of technical decision-making processes

Business Value

1. KYC Automation: Reduced manual processes taking over 10 minutes to under 2 minutes
2. Improved Accuracy: Created a more consistent and accurate authentication system than human judgment
3. Scalability: Architecture capable of handling multiple concurrent users
4. Cost Reduction: Decreased operational staffing and enabled 24/7 automated operation

KYC Processing Flow

🚀 Future Improvement Directions

1. Liveness Detection 🔒

Real-time facial movement detection to prevent photo/video spoofing attacks:

• Blink Detection: Natural eye movement patterns
• Head Movement Analysis: 3D rotation validation
• Challenge-Response: Random facial gesture requests

2. Mobile Optimization 📱

Native mobile apps for better camera control and user experience:

• iOS App: Native camera integration with ARKit
• Android App: Camera2 API with ML Kit acceleration
• Progressive Web App: Cross-platform fallback

3. Multi-national Document Support 🌍

Expand to support international ID documents:

• US Driver Licenses: All 50 states
• EU Passports: GDPR-compliant processing
• Asian ID Cards: Korea, Japan, China, Singapore

4. AI-based Quality Assessment 🤖

More sophisticated real-time quality evaluation:

• Advanced Blur Detection: Frequency domain analysis
• Lighting Optimization: Automatic exposure correction
• Face Pose Validation: 3D head pose estimation

5. Cloud Infrastructure ☁️

Scale globally with cloud deployment:

• AWS Multi-region: Low-latency global deployment
• Auto-scaling: Handle traffic spikes automatically
• CDN Integration: Fast media delivery worldwide

Through this project, I developed the capability to design and implement complex systems that create real business value, going beyond simple feature development. The experience of successfully integrating computer vision technologies in a web environment will be a great asset for my future development career.