Development and Testing

Relevant source files

• Base/Dockerfile
• tests/SeleniumTests/__init__.py
• tests/SmokeTests/__init__.py
• tests/bootstrap.sh
• tests/charts/ci/DeploymentAutoscaling-values.yaml
• tests/charts/ci/JobAutoscaling-values.yaml
• tests/charts/ci/NoAutoscaling-values.yaml
• tests/charts/ci/base-auth-ingress-values.yaml
• tests/charts/make/chart_cluster_setup.sh
• tests/charts/make/chart_test.sh
• tests/test.py

This page provides a guide for contributors to set up their development environment, run tests locally, and understand the testing framework. The Docker Selenium testing system validates browser automation, container orchestration, autoscaling, and grid readiness across multiple deployment scenarios.

For information about CI/CD workflows and automated releases, see CI/CD and Release Management. For details about the actual test suites executed in CI/CD pipelines, see Testing Framework and Test Suites. For contributing guidelines and development workflows, see Development Setup and Contributing.

Quick Start for Contributors

Prerequisites

• Docker Engine with Docker Compose
• Python 3.x with pip and virtualenv
• (Optional) Kubernetes cluster with Helm 3.x for chart testing

Running Tests Locally

Docker-level tests validate individual images and basic grid functionality:

Helm chart tests validate Kubernetes deployments:

Sources: tests/bootstrap.sh1-41 tests/charts/make/chart_test.sh1-571

Testing Framework Architecture

The Docker Selenium testing framework is built around Python unittest classes that validate browser functionality across different deployment modes and configurations. The framework supports parallel test execution, multiple browser versions, and various platform combinations.

Core Test Classes and Inheritance

Sources: tests/SeleniumTests/__init__.py65-400

Test Configuration and Environment Variables

SeleniumTests Configuration

The SeleniumTests module at tests/SeleniumTests/__init__.py17-41 uses environment variables to control test behavior:

Variable	Purpose	Default Value	Used At
SELENIUM_GRID_PROTOCOL	Grid protocol (http/https)	http	Line 17
SELENIUM_GRID_HOST	Grid hostname	localhost	Line 18
SELENIUM_GRID_PORT	Grid port	4444	Line 19
SELENIUM_GRID_USERNAME	Basic auth username	None	Line 20
SELENIUM_GRID_PASSWORD	Basic auth password	None	Line 21
CHART_CERT_PATH	TLS certificate path	None	Line 22
WEB_DRIVER_WAIT_TIMEOUT	WebDriver wait timeout (seconds)	60	Line 25
TEST_PARALLEL_HARDENING	Enable parallel test execution	false	Line 26
TEST_PARALLEL_COUNT	Number of parallel tests	5	Line 27
TEST_DELAY_AFTER_TEST	Delay after each test (seconds)	0	Line 28
TEST_NODE_RELAY	Test external relay nodes	false	Line 29
TEST_PLATFORMS	Target platform	linux/amd64	Line 31
TEST_MULTIPLE_VERSIONS	Test multiple browser versions	false	Line 35
TEST_MULTIPLE_PLATFORMS	Test multiple platforms	false	Line 36
TEST_ADD_CAPS_RECORD_VIDEO	Enable video recording capability	true	Line 33
TEST_CUSTOM_SPECIFIC_NAME	Test custom capabilities	false	Line 34

Chart Test Configuration

The chart_test.sh script at tests/charts/make/chart_test.sh5-72 provides Kubernetes-specific configuration:

Variable	Purpose	Default Value
CLUSTER_NAME	Kubernetes cluster name	chart-testing
RELEASE_NAME	Helm release name	test
SELENIUM_NAMESPACE	Target namespace	selenium
MATRIX_BROWSER	Browser type to test	NodeChrome
SELENIUM_GRID_AUTOSCALING	Enable autoscaling tests	true
SCALING_STRATEGY	KEDA scaling strategy	default
WAIT_TIMEOUT	Kubernetes wait timeout	90s
AUTOSCALING_POLL_INTERVAL	KEDA polling interval	20
AUTOSCALING_COOLDOWN_PERIOD	KEDA cooldown period	1800
LIMIT_RESOURCES	Apply resource limits	true
TEST_UPGRADE_CHART	Test chart upgrade	false
RENDER_HELM_TEMPLATE_ONLY	Only render templates	false

Sources: tests/SeleniumTests/__init__.py17-41 tests/charts/make/chart_test.sh5-72

Test Execution System

Docker-Level Test Execution

The test.py script at tests/test.py1-318 manages container lifecycle, network setup, and test orchestration.

Container Management and Test Orchestration Flow

Sources: tests/test.py241-318

Image Name and Test Class Mapping

The test.py script uses dictionaries to map test arguments to Docker image names and test classes:

IMAGE_NAME_MAP at tests/test.py61-82:

• Maps test names (e.g., NodeChrome) to image names (e.g., node-chrome)
• Handles browser-specific nodes and standalone images
• Supports all-browsers configurations

TEST_NAME_MAP at tests/test.py84-107:

• Maps image names to test class names
• Routes NodeChrome/StandaloneChrome → ChromeTests
• Routes NodeFirefox/StandaloneFirefox → FirefoxTests
• Routes NodeEdge/StandaloneEdge → EdgeTests

Sources: tests/test.py61-107

Image Building and Container Launch Process

The launch_container() function at tests/test.py170-221 builds Docker images on-demand and manages container lifecycle:

The function handles:

• Build Arguments: Passes FROM_IMAGE_ARGS with NAMESPACE, VERSION, BASE_VERSION at tests/test.py109-114
• Environment Variables: Sets proxy configuration and event bus settings at tests/test.py200-209
• Resource Limits: Configures shared memory (shm_size="2G") and read-only filesystem support at tests/test.py214-216

Sources: tests/test.py170-221

Helm Chart Test Execution

The chart_test.sh script at tests/charts/make/chart_test.sh1-571 orchestrates end-to-end Helm chart testing.

Chart Test Execution Flow

Sources: tests/charts/make/chart_test.sh1-571

Helm Command Construction

The script builds complex Helm commands by combining multiple configuration sources:

1. Base Values Files at tests/charts/make/chart_test.sh421-436:

   • base-auth-ingress-values.yaml: Authentication and ingress settings
   • base-recorder-values.yaml: Video recording configuration
   • base-resources-values.yaml: Resource limits (if enabled)
   • Browser-specific values (e.g., NodeChrome-values.yaml)

2. Image Configuration at tests/charts/make/chart_test.sh178-192:

   • Sets global.seleniumGrid.imageRegistry, imageTag
   • Configures autoscaling polling interval
   • Sets node max sessions and drain behavior

3. Feature Toggles via conditional --set flags:

   • TLS/HTTPS configuration at tests/charts/make/chart_test.sh346-369
   • Autoscaling strategy at tests/charts/make/chart_test.sh212-224
   • Basic authentication at tests/charts/make/chart_test.sh289-298
   • Monitoring/tracing at tests/charts/make/chart_test.sh240-250

Sources: tests/charts/make/chart_test.sh178-504

Local Development Environment Setup

Docker-Based Testing Setup

The Docker-level testing framework uses Python unittest with Selenium bindings to validate browser functionality. The setup process is automated through the bootstrap.sh script.

Bootstrap Process and Dependency Management

The bootstrap.sh script in tests/bootstrap.sh1-41 handles environment setup, dependency installation, and test execution:

Sources: tests/bootstrap.sh1-41

Required Python Dependencies

The testing framework requires the following Python packages (defined in tests/requirements.txt):

• selenium: WebDriver bindings for browser automation
• docker: Docker SDK for Python to manage container lifecycle
• requests: HTTP library for API calls to Grid endpoints
• Additional dependencies for test utilities

Environment Variables for Local Testing

Variable	Purpose	Default	Required
NAMESPACE	Docker registry namespace	-	Yes
VERSION	Image tag version	-	Yes
BINDING_VERSION	Selenium Python bindings version	-	Yes
SKIP_BUILD	Skip image building (use existing images)	false	No
PLATFORMS	Target platform for builds	linux/amd64	No
USE_RANDOM_USER_ID	Test with random user ID	false	No
FILESYSTEM_READ_ONLY	Test with read-only filesystem	false	No

Sources: tests/test.py43-54

Base Image Development Environment

The Base/Dockerfile provides the foundation for all images with development and runtime tools:

• Java Environment: OpenJDK 21 at Base/Dockerfile66
• Python Environment: Virtual environment at /opt/venv configured at Base/Dockerfile73-82
• Selenium Components: Pre-downloaded Selenium server JAR and OpenTelemetry dependencies at Base/Dockerfile126-153
• Development Tools: curl, jq, wget, supervisor for process management at Base/Dockerfile53-68

Sources: Base/Dockerfile1-234

Kubernetes/Helm Chart Testing Setup

The Helm chart testing framework validates Kubernetes deployments across multiple configurations and versions.

Chart Test Environment Setup

The chart_cluster_setup.sh script at tests/charts/make/chart_cluster_setup.sh1-66 sets up test Kubernetes clusters using either Kind or Minikube:

Sources: tests/charts/make/chart_cluster_setup.sh41-65

Chart Test Configuration

The chart_test.sh script at tests/charts/make/chart_test.sh1-571 orchestrates Helm chart deployment and testing with extensive configuration options:

Configuration	Purpose	Default
CLUSTER_NAME	Kubernetes cluster name	chart-testing
RELEASE_NAME	Helm release name	test
SELENIUM_NAMESPACE	Target namespace	selenium
MATRIX_BROWSER	Browser type to test	NodeChrome
SELENIUM_GRID_AUTOSCALING	Enable autoscaling tests	true
SCALING_STRATEGY	KEDA scaling strategy	default
TEST_EXISTING_KEDA	Use existing KEDA installation	false
TEST_MULTIPLE_VERSIONS	Test multiple browser versions	false

Sources: tests/charts/make/chart_test.sh5-72

Test Types and Coverage

Browser Test Suites

The framework implements comprehensive test coverage across multiple browsers and scenarios:

Generic Test Cases

All browser tests inherit from SeleniumGenericTests and execute:

• Page Navigation: test_title() validates basic page loading and title verification
• Frame Handling: test_with_frames() tests nested frame navigation
• Form Interaction: test_select_from_a_dropdown() validates dropdown selection
• Authentication: test_visit_basic_auth_secured_page() tests HTTP basic auth
• Media Playback: test_play_video() validates video element interaction
• File Downloads: test_download_file() tests download functionality with managed downloads

Sources: tests/SeleniumTests/__init__.py67-130

Browser-Specific Test Implementation

Each browser test class implements a setUp() method that configures WebDriver options and capabilities:

ChromeTests setUp() at tests/SeleniumTests/__init__.py146-203:

FirefoxTests setUp() at tests/SeleniumTests/__init__.py255-305:

EdgeTests setUp() at tests/SeleniumTests/__init__.py206-251:

• Similar structure to ChromeTests
• Uses EdgeOptions() instead of ChromeOptions()
• Supports same capability configuration pattern

All browser tests support:

• Video Recording: se:recordVideo capability controlled by TEST_ADD_CAPS_RECORD_VIDEO
• Test Naming: se:name set to {testMethodName} ({className})
• Screen Resolution: se:screenResolution set to 1920x1080
• Multiple Versions: Random selection from version lists when TEST_MULTIPLE_VERSIONS=true
• Platform Testing: Random platform selection when TEST_MULTIPLE_PLATFORMS=true

Sources: tests/SeleniumTests/__init__.py145-305

Smoke Tests and Grid Validation

The SmokeTests module at tests/SmokeTests/__init__.py1-75 provides grid readiness validation before executing Selenium tests.

Grid Status Validation Flow

SmokeTests Implementation Details

The GridTest class at tests/SmokeTests/__init__.py70-75 validates:

1. Grid Readiness via smoke_test_container() at tests/SmokeTests/__init__.py26-61:

   • Polls /status endpoint with retry logic
   • Validates HUB_CHECKS_MAX_ATTEMPTS and HUB_CHECKS_INTERVAL
   • Handles autoscaling scenarios where min_replica=0

2. TLS Certificate Validation via client_verify_cert() at tests/SmokeTests/__init__.py62-68:

   • Uses REQUESTS_CA_BUNDLE environment variable
   • Verifies certificate against CHART_CERT_PATH
   • Validates HTTPS connections with HTTPBasicAuth

Sources: tests/SmokeTests/__init__.py1-75

Autoscaling and Parallel Test Execution

The framework supports autoscaling validation through concurrent test execution:

Sources: tests/SeleniumTests/__init__.py327-369