DEV Community

HariOm
HariOm

Posted on

Test Automation Architecture: Data Management, Execution & Orchestration in Hybrid Environments - Part 3 of 4

#devops #architecture #testautomation #designpatterns

Introduction

In Part 1, we explored the architectural problem space and why manual test operations fail at scale. In Part 2, we introduced the complete system architecture and detailed Phases 1 and 1.5—the foundation that enables everything else.

Quick Recap:

  • Phase 1: Feature testing in cloud environments with automatic tagging
  • Phase 1.5: Release tagging validation—the critical gate that prevents consolidation problems

Now we're ready for the transformation phases:

  • Phase 2: Data Consolidation (3-4 days to 4-8 hours)
  • Phase 3: Release Testing (two-tier strategy in PreProd)
  • Phase 4: Production Deployment (synchronized code and data)
  • Phase 5: Continuous Growth (cumulative regression)

This article details how these phases work and how they deliver the promised transformation.

Table of Contents

Phase 2: Data Consolidation

Overview

Duration: 4-8 hours (down from 3-4 days manual)

Location: Data Consolidation Service

Input: Tagged and validated data from all ST environments

Output: Merged dataset in PreProd Staging Database

Automation: 70-80% conflicts auto-resolved

This is where the architecture delivers its biggest time savings. What took 3-4 days of manual work now takes 4-8 hours of mostly automated processing.

The Consolidation Process 

Phase 1.5 Complete (Data Tagged & Locked) ↓ Consolidation Service Starts ↓ Collect Data from ST-1, ST-2, ... ST-N ↓ Detect Conflicts (Duplicates, etc.) ↓ ┌────┴────┐ │ │ Simple Complex (70-80%) (20-30%) │ │ Auto-Resolve Manual Review │ │ └────┬────┘ ↓ Validation (Integrity Checks) ↓ ┌───────┴───────┐ │ │ Valid? Invalid? │ │ ↓ ↓ Load to Fix Issues PreProd & Retry Staging │ │ │ └──────┬───────┘ ↓ Generate Audit Trail ↓ Consolidation Complete
Enter fullscreen mode Exit fullscreen mode

Step-by-Step Process

Step 1: Collection (30 minutes)

Data Consolidation Service connects to all ST databases:

  • Queries: SELECT * FROM test_data WHERE release = 'R2'
  • Collects only data tagged with current release
  • Extracts with metadata intact
  • Transfers to consolidation staging area

What's collected: Users, accounts, transactions, configurations, test scenarios—everything tagged for this release.

Step 2: Conflict Detection (1-2 hours)

Service scans for conflicts across datasets:

Types of conflicts detected:

  1. Duplicate Primary Keys

    • Same user_id exists in ST-1 and ST-3
    • Same account_id in ST-2 and ST-5

  2. Duplicate Natural Keys

  3. Conflicting Attributes

    • Same email but different roles (Admin vs. User)
    • Same account but different balances

  4. Relationship Conflicts

    • Account references user that doesn't exist in consolidated set
    • Transaction references account with mismatched data

Example conflict:

Conflict ID: C-001 Type: Duplicate Email with Conflicting Attributes ST-1 Record: email: testuser1@company.com role: Administrator balance: $0 feature: FEAT-101 created: 2024-01-10 ST-3 Record: email: testuser1@company.com role: Customer balance: $10,000 feature: FEAT-105 created: 2024-01-12
Enter fullscreen mode Exit fullscreen mode

Step 3: Conflict Classification (30 minutes)

Service classifies each conflict by complexity:

Simple Conflicts (Auto-resolvable):

  • Clear temporal priority (newest wins)
  • Clear feature priority (critical feature wins)
  • Exact duplicates (identical data, just keep one)
  • Non-conflicting merges (different attributes, can merge)

Complex Conflicts (Requires human review):

  • Conflicting core attributes (role, permissions)
  • Business logic dependencies
  • Multiple features with equal priority
  • Data integrity concerns

Conflict Detection and Resolution

Auto-Resolution Rules (70-80% of conflicts)

The system applies systematic rules:

Rule 1: Temporal Priority

IF conflict_type == "duplicate" AND attributes_identical: keep_newer_record() reason = "Newer timestamp"
Enter fullscreen mode Exit fullscreen mode

Rule 2: Feature Priority

IF conflict_type == "attribute_mismatch": feature_priority = get_priority_from_release_plan() IF feature_A.priority > feature_B.priority: keep_feature_A_data() reason = "Higher priority feature"
Enter fullscreen mode Exit fullscreen mode

Rule 3: Environment Precedence

IF conflict_type == "duplicate" AND no_clear_priority: # Use predefined environment precedence keep_from_primary_environment() reason = "Environment precedence rule"
Enter fullscreen mode Exit fullscreen mode

Rule 4: Non-Conflicting Merge

IF attributes_dont_conflict: merge_attributes() reason = "Complementary data merged"
Enter fullscreen mode Exit fullscreen mode

Rule 5: Exact Duplicate

IF records_identical: keep_one_copy() reason = "Exact duplicate removed"
Enter fullscreen mode Exit fullscreen mode

Example Auto-Resolution:

Conflict: testuser2@company.com exists in ST-1 and ST-4 Analysis: - Both created for same feature (FEAT-102) - Identical attributes except timestamp - ST-4 record is newer (2024-01-12 vs 2024-01-10) Resolution: AUTOMATIC - Keep ST-4 record (newer) - Discard ST-1 record - Audit trail: "Duplicate resolved via temporal priority"
Enter fullscreen mode Exit fullscreen mode

The Manual Review Process

For the 20-30% that can't auto-resolve:

Step 1: Queue Generation

System creates structured review queue:

Manual Review Queue - Release R2 Total conflicts: 85 Auto-resolved: 62 (73%) Requires review: 23 (27%) Priority 1 (Critical): 5 conflicts Priority 2 (High): 10 conflicts Priority 3 (Medium): 8 conflicts
Enter fullscreen mode Exit fullscreen mode

Step 2: Review Interface

QA Lead opens Review Interface showing:

  • Side-by-side comparison of conflicting records
  • Context: Which features, which environments, who created
  • Related data (dependencies, relationships)
  • Suggested resolution with rationale
  • Options: Keep A, Keep B, Keep Both (rename), Merge, Delete Both

Step 3: Decision Making

For each conflict, reviewer:

  1. Reviews context: Reads feature descriptions, test scenarios
  2. Consults if needed: Asks feature owners or testers
  3. Makes decision: Selects resolution option
  4. Documents rationale: Explains why (for audit trail)

Example Manual Review:

Conflict C-015: testuser_merchant@company.com Priority: HIGH ST-2 Record (FEAT-104 - Merchant Payments): role: Merchant merchant_id: M-1001 balance: $50,000 payment_gateway: enabled ST-5 Record (FEAT-108 - Merchant Refunds): role: Merchant merchant_id: M-1001 balance: $25,000 refund_capability: enabled System Analysis: Cannot auto-resolve - Same merchant_id but different balances - Different feature requirements - Both may be needed for different test scenarios Reviewer Decision: KEEP BOTH (Rename) - Rename ST-2: testuser_merchant_payments@company.com - Rename ST-5: testuser_merchant_refunds@company.com - Rationale: "Both scenarios needed for comprehensive testing" - Approved by: qa.lead@company.com - Timestamp: 2024-01-15T16:45:00Z
Enter fullscreen mode Exit fullscreen mode

Step 4: Apply Decisions

Once all reviews complete:

  • System applies manual decisions
  • Updates audit trail
  • Proceeds to validation

Time Investment:

  • Auto-resolution: Instant (handled by rules)
  • Manual review: 15-30 minutes per complex conflict
  • 23 conflicts × 20 min average = approximately 8 hours
  • Can be distributed across team

Compare to manual consolidation:

  • Manual: 3-4 days (24-32 hours) for ONE person
  • Automated: 4-8 hours total, parallelizable across team

Step 4: Validation (1 hour)

After resolution, service validates:

Integrity Checks:

  • All foreign keys resolve correctly
  • No orphaned records
  • Relationship chains intact
  • Data types valid
  • Required fields populated

Business Rule Validation:

  • Account balances reasonable
  • User roles valid
  • Permissions consistent
  • Test scenarios complete

If validation fails:

  • Issues flagged
  • Fixes applied
  • Re-validate
  • Iterate until clean

Step 5: Loading (30 minutes)

Once validated:

  • Export consolidated dataset
  • Transfer to on-premises environment (secure protocols)
  • Import into PreProd Staging Database
  • Verify record counts and checksums
  • Confirm data loaded correctly

Step 6: Audit Trail Generation

System generates complete audit trail:

Consolidation Audit Trail - Release R2 Completed: 2024-01-15T18:00:00Z Summary: - Total records collected: 425 - Conflicts detected: 85 - Auto-resolved: 62 (73%) - Manual review: 23 (27%) - Final dataset: 390 records - Records merged: 15 - Records renamed: 10 - Records deleted: 20 Time Breakdown: - Collection: 28 minutes - Conflict detection: 1.5 hours - Auto-resolution: Instant - Manual review: 7.2 hours (distributed) - Validation: 52 minutes - Loading: 25 minutes Total elapsed time: 6.4 hours All decisions documented and traceable.
Enter fullscreen mode Exit fullscreen mode

Phase 3: Release Testing

Overview

Duration: 1-2 days

Location: On-Premises PreProd Environment

Objective: Validate consolidated release before production

Strategy: Two-tier testing approach

The Two-Tier Testing Strategy

Phase 3 uses a two-tier approach that balances speed with comprehensiveness:

┌─────────────────────────────────────────────────────┐ │ TIER 1: STAGING DATABASE │ ├─────────────────────────────────────────────────────┤ │ │ │ • New Release Data (This Release Only) │ │ • Incremental Testing (New Features) │ │ • Fast Feedback (2-4 hours) │ │ │ └──────────────────┬──────────────────────────────────┘ │ ↓ (if Pass) ┌─────────────────────────────────────────────────────┐ │ TIER 2: MASTER DATABASE │ ├─────────────────────────────────────────────────────┤ │ │ │ • Cumulative Data (All Previous Releases) │ │ • Full Regression (All Test Cases) │ │ • Comprehensive (8-12 hours) │ │ │ └─────────────────────────────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

Why Two Tiers?

Tier 1 (Staging) gives fast feedback on new features:

  • Tests only new functionality
  • Catches major issues quickly
  • Allows fast iteration if problems found
  • 2-4 hours to complete

Tier 2 (Master) provides comprehensive validation:

  • Tests everything (new + old features)
  • Catches regression issues
  • Validates cumulative behavior
  • 8-12 hours to complete (but runs overnight)

This strategy prevents the common problem: spending days on full regression only to find a basic issue that could have been caught in hours.

Testing Workflow

Stage 1: Staging Database Testing (2-4 hours)

Step 1: Load New Data

  • Consolidated data loaded into PreProd Staging Database
  • Only contains data for current release (R2)
  • Clean slate for incremental testing

Step 2: Deploy Release Code

  • Release code deploys to PreProd environment
  • Points to Staging Database
  • Configuration validated

Step 3: Execute Incremental Tests

  • Run Orchestrator triggers test suite
  • Executes tests for new features only:
    • FEAT-101, FEAT-102, ... FEAT-108
  • Parallel execution across test agents
  • Results aggregate in real-time

Test Categories:

  • Smoke tests (critical paths)
  • Feature-specific tests
  • Integration tests for new features
  • API contract tests
  • UI tests for new screens

Time: 2-4 hours depending on feature count

Decision Point:

IF Tier 1 tests PASS: Proceed to Tier 2 (Master Database testing) ELSE: - Log defects - Fix issues - Re-consolidate if data problems - Re-test DO NOT proceed to Tier 2 until Tier 1 passes
Enter fullscreen mode Exit fullscreen mode

Why this matters: No point running 8-12 hours of comprehensive testing if basic features are broken. Fix fast, then validate comprehensively.

Stage 2: Master Database Testing (8-12 hours)

Step 1: Merge to Master Database

  • Consolidated data from Staging merges into Master Database
  • Master Database contains:
    • Current release data (R2)
    • All previous releases (R1, R0, etc.)
    • Cumulative test knowledge
  • This is the growing knowledge base

Step 2: Point to Master Database

  • PreProd environment reconfigured
  • Points to Master Database
  • Code remains same (release code)

Step 3: Execute Full Regression

  • Run Orchestrator triggers comprehensive test suite
  • Executes ALL test cases:
    • Current release tests (FEAT-101 to FEAT-108)
    • Previous release tests (all features from R1, R0, etc.)
    • Cross-release integration tests
    • End-to-end scenarios
    • Performance tests
    • Security tests

Test Execution:

  • Parallel execution across available agents
  • Typically runs overnight
  • 500-1000+ test cases
  • 8-12 hours depending on suite size

What This Validates:

  • New features work
  • Old features still work (no regression)
  • New features don't break old features
  • Cross-feature interactions work
  • System performs at scale

Step 4: Results Analysis

Team reviews results:

Regression Test Results - Release R2 Total Tests: 847 Passed: 831 (98.1%) Failed: 12 (1.4%) Skipped: 4 (0.5%) New Features (Tier 1): - Tests: 125 - Passed: 123 (98.4%) - Failed: 2 Regression (Previous Releases): - Tests: 722 - Passed: 708 (98.1%) - Failed: 10 - New regressions: 3 (needs investigation)
Enter fullscreen mode Exit fullscreen mode

Step 5: Defect Resolution

If failures found:

  • Log defects in tracking system
  • Prioritize by severity
  • Fix critical issues
  • Re-test after fixes
  • Iterate until acceptable pass rate (typically 99%+)

Step 6: Release Approval

Once tests pass:

  • QA Lead approves release
  • Generates test report for stakeholders
  • Release marked as "Ready for Production"

Phase 4: Production Deployment

Overview

Duration: approximately 1 day (including validation)

Location: On-Premises Production

Objective: Deploy release safely with rollback capability

Synchronized Deployment

The key architectural principle: Code and data deploy together, rollback together.

Release Approved ↓ Prepare Deployment Packages ↓ ┌───┴───┐ │ │ Code Data Package Package │ │ └───┬───┘ ↓ Version Lock (Release R2) ↓ Deploy to Production (Synchronized) ↓ ┌───┴───┐ │ │ Deploy Deploy Code Data │ │ └───┬───┘ ↓ Validation ↓ Tests Pass? ↓ ┌───┴───┐ │ │ Yes No │ │ Deploy Synchronized Complete Rollback │ ┌───┴───┐ │ │ Rollback Rollback Code Data │ │ └───┬───┘ ↓ Previous Stable State
Enter fullscreen mode Exit fullscreen mode

Deployment Process

Step 1: Pre-Deployment Preparation

Code Package:

  • Application binaries
  • Configuration files
  • Database migration scripts
  • Deployment scripts

Data Package:

  • Test data export from PreProd Master
  • Data migration scripts
  • Validation scripts

Versioning:

  • Code: Tagged with release version (R2)
  • Data: Tagged with same release version (R2)
  • Both linked: Ensures consistency

Step 2: Deployment Window

Typically during low-traffic period:

  • Schedule maintenance window
  • Notify stakeholders
  • Put system in maintenance mode (if required)

Step 3: Synchronized Deployment

Sequence:

  1. Backup current state (both code and data)
  2. Deploy code to production servers
  3. Deploy data to production database
  4. Run migration scripts (if schema changes)
  5. Restart services
  6. Remove maintenance mode

Critical: Both code and data deploy as atomic operation. If either fails, both rollback.

Step 4: Production Validation

Automated validation suite runs:

Smoke Tests (5-10 minutes):

  • Critical paths functional
  • Users can log in
  • Core transactions work
  • APIs responding
  • Database connectivity

Production Verification Tests (30 minutes):

  • End-to-end scenarios
  • Key workflows
  • Integration points
  • Performance checks

Monitoring (Continuous):

  • Error rates
  • Response times
  • System health metrics
  • User activity patterns

Step 5: Go/No-Go Decision

If validation passes:

  • Deployment marked successful
  • Monitoring continues
  • Team on standby for first few hours
  • Release complete

If validation fails:

  • Immediate rollback initiated
  • See next section

Rollback Capability

This is the critical safety net. The architecture ensures you can always rollback safely.

Why Synchronized Rollback Matters:

Traditional problem:

Code deployed: Version 2.0 (expects new data structure) Data deployed: Failed or incomplete Result: Code looking for data that doesn't exist - System breaks Manual fix required - Hours of downtime
Enter fullscreen mode Exit fullscreen mode

Synchronized approach:

Code deployed: Version 2.0 Data deployed: Version 2.0 Validation: FAIL Rollback triggered: - Code reverts to 1.9 - Data reverts to 1.9 - Both synchronized Result: System returns to known good state Downtime: Minutes, not hours
Enter fullscreen mode Exit fullscreen mode

Rollback Process:

Step 1: Detect Failure

  • Validation tests fail
  • Monitoring alerts trigger
  • Manual intervention

Step 2: Initiate Rollback

  • Rollback command issued
  • Both code and data rollback triggered

Step 3: Code Rollback

  • Revert to previous version (R1)
  • Redeploy previous application binaries
  • Restore previous configuration

Step 4: Data Rollback

  • Revert database to previous state
  • Can use:
    • Database backup restoration
    • Transaction log replay
    • Point-in-time recovery
  • Ensures data matches previous code version

Step 5: Validation

  • Quick smoke tests
  • Confirm system functional
  • Verify data integrity

Step 6: Post-Mortem

  • Analyze what went wrong
  • Fix issues
  • Re-test in PreProd
  • Schedule re-deployment

Time to Rollback: 10-30 minutes (vs hours of manual recovery)

Phase 5: Continuous Growth

Overview

Duration: Ongoing (after production deployment)

Location: Production

Objective: Continuous regression with expanding coverage

This is where the cumulative knowledge model delivers long-term value.

Nightly Cumulative Regression

Every Night:

  • Run Orchestrator triggers full regression suite
  • Executes against Master Database (cumulative knowledge)
  • Tests ALL features from ALL releases
  • Results available next morning

What This Provides:

Early Detection:

  • Production issues caught overnight
  • Not waiting for next release cycle
  • Proactive vs reactive

Confidence:

  • Continuous validation
  • Always know system health
  • Regression caught immediately

Coverage Growth:

  • Each release adds tests
  • Suite grows automatically
  • Regression becomes more comprehensive over time

Coverage Growth Over Time

Traditional approach:

Release 1: 500 test cases Release 2: 500 test cases (recreated from scratch) Release 3: 500 test cases (recreated from scratch) Coverage: Flat, doesn't grow
Enter fullscreen mode Exit fullscreen mode

Cumulative approach:

Release 1: 500 test cases - Master DB Release 2: +150 test cases - Master DB (total: 650) Release 3: +200 test cases - Master DB (total: 850) Release 4: +100 test cases - Master DB (total: 950) Coverage: Growing continuously After 2 years: 1500+ test cases
Enter fullscreen mode Exit fullscreen mode

Why This Works:

Test Data Preserved:

  • Data not recreated each release
  • Historical scenarios remain
  • Coverage accumulates

No Deletion:

  • Consolidation doesn't delete data to avoid conflicts
  • Conflicts resolved systematically
  • All valid scenarios kept

Automatic Execution:

  • Nightly runs exercise all accumulated tests
  • No manual effort to maintain
  • Scales automatically

Performance Management

As suite grows, manage performance:

Database Partitioning:

  • Partition Master Database by release
  • Query optimization for large datasets
  • Indexing strategy

Selective Execution:

  • Full regression nightly (all tests)
  • Smoke tests on-demand (critical tests)
  • Feature-specific on commits (relevant tests)

Archival Strategy:

  • Archive very old releases (greater than 2 years)
  • Move to cold storage
  • Can restore if needed

Test Pruning:

  • Periodically review obsolete tests
  • Remove tests for deprecated features
  • Keep coverage lean and relevant

The Complete Lifecycle in Action

Let's see how a release flows through all phases:

Release Cycle Timeline - R2 Day 1-14: ████████████████ Phase 1: Feature Testing (Cloud) Day 15: █ Phase 1.5: Release Tagging (1 hour) Day 15: ██ Phase 2: Data Consolidation (6 hours) Day 16: ████ Phase 3: Tier 1 Staging (4 hours) Day 16: ████████ Phase 3: Tier 2 Master (12 hours) Day 17: ███ Phase 4: Deployment (4 hours) Day 17: █ Phase 4: Validation (2 hours) Day 18+: ████████████████████ Phase 5: Continuous Regression Total: ~17 days (was ~21 days with manual consolidation)
Enter fullscreen mode Exit fullscreen mode

Timeline:

  • Days 1-14: Feature development and testing (Phase 1)
  • Day 15 morning: Release tagging validation (Phase 1.5)
  • Day 15 afternoon: Data consolidation (Phase 2)
  • Day 16: Release testing (Phase 3)
  • Day 17: Production deployment (Phase 4)
  • Day 18+: Continuous regression (Phase 5)

Total Time: approximately 17 days (was approximately 21 days with manual consolidation)

Key Improvements:

  • 3-4 days saved (consolidation automation)
  • Higher quality (comprehensive testing)
  • Lower risk (rollback capability)
  • Growing coverage (cumulative knowledge)

What's Coming Next

This article detailed Phases 2-5: the implementation that delivers the transformation.

In Part 4 (final article), we'll cover:

Real-World Metrics:

  • Actual time savings (before/after comparison)
  • Cost reduction (QA productivity, infrastructure)
  • Quality improvement (defect rates, rollback rates)
  • Coverage growth (actual numbers over time)

Implementation Strategy:

  • Phase-by-phase rollout (10-17 month timeline)
  • Prerequisites and dependencies
  • Team training and adoption
  • Common challenges and solutions

Trade-offs and Limitations:

  • When this architecture makes sense
  • When it's overkill
  • Known limitations and workarounds
  • Alternative approaches

Applicability Guidance:

  • Assessment framework (is this right for you?)
  • Decision criteria
  • Migration strategies
  • Success metrics

About This Series

This architectural pattern is part of HariOm-Labs, an open-source initiative focused on solving real Cloud DevOps and Platform Engineering challenges with production-grade solutions.

The mission: Share technically rigorous, production-ready implementations with comprehensive documentation of trade-offs, architectural decisions, and real-world considerations. Not toy examples, but battle-tested patterns that teams can actually use.

GitHub: https://github.com/HariOm-Labs

Have you implemented similar consolidation or testing patterns? What challenges did you face with data management across environments? Share your experiences in the comments!

Top comments (0)