Testing Framework and Test Suites

Relevant source files

• Base/Dockerfile
• tests/SeleniumTests/__init__.py
• tests/SmokeTests/__init__.py
• tests/bootstrap.sh
• tests/test.py

Purpose and Scope

This page documents the Docker-level testing framework used to validate Selenium Grid Docker images. The framework consists of a Python-based test orchestrator (test.py) that manages container lifecycle, environment setup scripts (bootstrap.sh), and comprehensive test suites (SmokeTests, SeleniumTests) that verify Grid functionality and browser automation capabilities.

For Kubernetes/Helm testing, see Chart Testing and Validation. For CI/CD workflows that execute these tests, see Build and Test Workflows.

---

Testing Framework Architecture

The testing framework operates in three layers: environment setup, container orchestration, and test execution. The framework can test both standalone containers and distributed Hub-Node configurations, building images on-demand if needed.

Sources: tests/bootstrap.sh1-41 tests/test.py1-318 tests/SeleniumTests/__init__.py1-400 tests/SmokeTests/__init__.py1-75

---

Bootstrap Script (bootstrap.sh)

The bootstrap.sh script serves as the entry point for test execution, handling Python environment setup and dependency installation before delegating to test.py.

Execution Flow

Sources: tests/bootstrap.sh1-41

Key Responsibilities

Responsibility	Implementation
Environment Isolation	Creates Python virtualenv (docker-selenium-tests) for local runs tests/bootstrap.sh9-11
Dependency Management	Installs Selenium bindings matching BINDING_VERSION or latest snapshot tests/bootstrap.sh14-19
Test Requirements	Installs packages from requirements.txt tests/bootstrap.sh21
Test Routing	Routes autoscaling tests to unittest directly, others to test.py tests/bootstrap.sh23-32
CI Detection	Skips virtualenv creation when CI=false to use pre-existing environment tests/bootstrap.sh8-12

Sources: tests/bootstrap.sh1-41

---

Test Orchestrator (test.py)

The test.py script is the core orchestrator that manages Docker container lifecycle and coordinates test execution. It supports both standalone and distributed Hub-Node deployments.

Container Lifecycle Management

Sources: tests/test.py241-318

Image Building Logic

The orchestrator builds images on-demand unless SKIP_BUILD=true. The launch_container() function handles multi-platform builds:

Key Functions:

• set_from_image_base_for_standalone(container): For Standalone* images, extracts browser name (e.g., StandaloneChrome → node-chrome) and sets FROM_IMAGE_ARGS['BASE'] tests/test.py223-227
• get_build_path(container): Converts Standalone* container names to their build paths (e.g., StandaloneChrome → Standalone) tests/test.py229-235
• standalone_browser_container_matches(container): Regex pattern (Standalone)(Chromium|Chrome|Firefox|Edge) tests/test.py237-238

Sources: tests/test.py170-221 tests/test.py223-238

Container Name and Image Mappings

The orchestrator uses two key mappings to translate test targets to Docker resources:

IMAGE_NAME_MAP tests/test.py61-82:

TEST_NAME_MAP tests/test.py84-107:

Sources: tests/test.py61-107

Hub and Node Launching

Hub Launch Flow (launch_hub()) tests/test.py127-159:

1. Check for existing selenium-hub container
2. Kill and remove if found
3. Launch new Hub container with ports {4442, 4443, 4444}
4. Optionally apply random user ID if USE_RANDOM_USER_ID=true

Node Launch Flow (via launch_container()) tests/test.py170-221:

1. Set environment variables: SE_EVENT_BUS_HOST, SE_EVENT_BUS_PUBLISH_PORT, SE_EVENT_BUS_SUBSCRIBE_PORT
2. Enable managed downloads: SE_NODE_ENABLE_MANAGED_DOWNLOADS=true
3. Launch with shm_size=2G (required for browser memory)
4. Apply read-only filesystem if FILESYSTEM_READ_ONLY=true with /tmp as tmpfs

Sources: tests/test.py127-159 tests/test.py170-221

---

Smoke Tests (SmokeTests)

Smoke tests provide basic health checks by verifying Grid readiness via the /status endpoint.

GridTest Class

Health Check Algorithm

The smoke_test_container(port) method implements a retry mechanism with configurable attempts tests/SmokeTests/__init__.py26-60:

Environment Variables:

Variable	Default	Purpose
HUB_CHECKS_MAX_ATTEMPTS	3	Maximum retry attempts tests/SmokeTests/__init__.py18
HUB_CHECKS_INTERVAL	10	Seconds between retries tests/SmokeTests/__init__.py19
SELENIUM_GRID_AUTOSCALING	false	Enable autoscaling-aware assertions tests/SmokeTests/__init__.py16
SELENIUM_GRID_AUTOSCALING_MIN_REPLICA	0	Expected minimum replicas tests/SmokeTests/__init__.py17

Sources: tests/SmokeTests/__init__.py1-75

TLS Certificate Verification

When SELENIUM_GRID_PROTOCOL=https, the client_verify_cert() method validates certificate trust tests/SmokeTests/__init__.py62-67:

The REQUESTS_CA_BUNDLE environment variable is set from CHART_CERT_PATH tests/SmokeTests/__init__.py21-22

Sources: tests/SmokeTests/__init__.py62-67 tests/SmokeTests/__init__.py21-22

---

Selenium Tests (SeleniumTests)

The SeleniumTests module contains comprehensive browser automation tests organized into a class hierarchy.

Test Class Hierarchy

Sources: tests/SeleniumTests/__init__.py65-400

SeleniumGenericTests Base Class

The SeleniumGenericTests class defines common test methods inherited by browser-specific classes tests/SeleniumTests/__init__.py65-143:

Test Method	Purpose	Key Assertions
test_title()	Verify page navigation and title	driver.title == 'The Internet' tests/SeleniumTests/__init__.py67-71
test_with_frames()	Test iframe handling	Find element in nested frames tests/SeleniumTests/__init__.py74-80
test_select_from_a_dropdown()	Test dropdown selection	Verify selected option tests/SeleniumTests/__init__.py83-96
test_visit_basic_auth_secured_page()	HTTP Basic Auth	Navigate to http://admin:admin@... tests/SeleniumTests/__init__.py99-103
test_play_video()	Video playback	Verify video paused == False tests/SeleniumTests/__init__.py105-115
test_download_file()	File download (managed downloads)	Verify get_downloadable_files() contains expected file tests/SeleniumTests/__init__.py117-129

Common Patterns:

• All tests use WebDriverWait with configurable timeout (WEB_DRIVER_WAIT_TIMEOUT, default 60s) tests/SeleniumTests/__init__.py25
• Tests target public test sites: the-internet.herokuapp.com, googleads.github.io
• tearDown() quits the driver with optional delay (TEST_DELAY_AFTER_TEST) tests/SeleniumTests/__init__.py131-142

Sources: tests/SeleniumTests/__init__.py65-143

Browser-Specific Test Classes

Each browser class implements setUp() to configure browser-specific options:

ChromeTests Setup tests/SeleniumTests/__init__.py145-203

Key Configurations:

• Download Management: Controls whether to use Selenium 4 managed downloads or legacy directory-based downloads tests/SeleniumTests/__init__.py149-151
• Video Recording: Sets se:recordVideo capability when enabled tests/SeleniumTests/__init__.py152-153
• Custom Capabilities: Supports arbitrary test capabilities for validation tests/SeleniumTests/__init__.py154-156
• Headless Mode: Adds --headless=new argument tests/SeleniumTests/__init__.py159-160
• Version Matrix Testing: Randomly selects from LIST_CHROMIUM_VERSIONS tests/SeleniumTests/__init__.py161-165
• Android/Appium Support: Configures Appium capabilities for Android relay nodes tests/SeleniumTests/__init__.py166-176

Sources: tests/SeleniumTests/__init__.py145-203

FirefoxTests Setup tests/SeleniumTests/__init__.py254-305

Firefox setup follows similar patterns but includes Firefox-specific profile configuration:

Firefox also includes two unique tests:

• test_title_and_maximize_window(): Tests window maximization tests/SeleniumTests/__init__.py307-310
• test_accept_languages(): Validates locale settings and optional language pack installation tests/SeleniumTests/__init__.py312-324

Sources: tests/SeleniumTests/__init__.py254-324

EdgeTests Setup tests/SeleniumTests/__init__.py205-251

EdgeTests uses EdgeOptions() and follows the same configuration pattern as ChromeTests, supporting all the same feature flags and capabilities.

Sources: tests/SeleniumTests/__init__.py205-251

Client Configuration

All tests use a shared CLIENT_CONFIG object for WebDriver connection settings tests/SeleniumTests/__init__.py52-59:

Sources: tests/SeleniumTests/__init__.py52-59

Autoscaling Tests

The Autoscaling class provides parallel test execution for KEDA-based autoscaling validation tests/SeleniumTests/__init__.py327-369:

Key Features:

• Concurrent Execution: Uses ThreadPoolExecutor for parallel test execution tests/SeleniumTests/__init__.py329
• Mixed Shuffling: Randomizes test order across browser types tests/SeleniumTests/__init__.py337
• Automatic Retry: Failed tests are automatically rerun once tests/SeleniumTests/__init__.py356-368
• Timing Metrics: Tracks test completion time tests/SeleniumTests/__init__.py346

Test Classes Using Autoscaling:

• DeploymentAutoscalingTests: Tests KEDA ScaledObject autoscaling tests/SeleniumTests/__init__.py371-378
• JobAutoscalingTests: Tests KEDA ScaledJob autoscaling tests/SeleniumTests/__init__.py381-388

Both delegate to TestPlatform.add_test_based_platform(repeat) to generate test lists based on TEST_PLATFORMS and TEST_PARALLEL_COUNT tests/SeleniumTests/__init__.py391-399

Sources: tests/SeleniumTests/__init__.py327-399

---

Configuration and Environment Variables

The test framework is highly configurable through environment variables:

Grid Connection Settings

Variable	Default	Purpose
SELENIUM_GRID_PROTOCOL	http	Protocol (http/https) tests/SeleniumTests/__init__.py17
SELENIUM_GRID_HOST	localhost	Grid hostname tests/SeleniumTests/__init__.py18
SELENIUM_GRID_PORT	4444	Grid port tests/SeleniumTests/__init__.py19
SELENIUM_GRID_USERNAME	None	Basic auth username tests/SeleniumTests/__init__.py20
SELENIUM_GRID_PASSWORD	None	Basic auth password tests/SeleniumTests/__init__.py21
CHART_CERT_PATH	None	TLS certificate path tests/SeleniumTests/__init__.py22

Sources: tests/SeleniumTests/__init__.py17-22

Test Behavior Settings

Variable	Default	Purpose
SELENIUM_GRID_TEST_HEADLESS	false	Run browsers in headless mode tests/SeleniumTests/__init__.py23
SELENIUM_ENABLE_MANAGED_DOWNLOADS	true	Use Selenium 4 managed downloads tests/SeleniumTests/__init__.py24
WEB_DRIVER_WAIT_TIMEOUT	60	WebDriverWait timeout (seconds) tests/SeleniumTests/__init__.py25
TEST_DELAY_AFTER_TEST	0	Delay before driver.quit() tests/SeleniumTests/__init__.py28
TEST_ADD_CAPS_RECORD_VIDEO	true	Enable video recording capability tests/SeleniumTests/__init__.py33
TEST_CUSTOM_SPECIFIC_NAME	false	Test custom capabilities tests/SeleniumTests/__init__.py34

Sources: tests/SeleniumTests/__init__.py23-34

Matrix Testing Settings

Variable	Default	Purpose
TEST_MULTIPLE_VERSIONS	false	Enable browser version matrix testing tests/SeleniumTests/__init__.py35
TEST_MULTIPLE_PLATFORMS	false	Enable platform matrix testing tests/SeleniumTests/__init__.py36
TEST_MULTIPLE_PLATFORMS_RELAY	false	Enable relay platform testing (SauceLabs) tests/SeleniumTests/__init__.py37
TEST_MULTIPLE_VERSIONS_EXPLICIT	true	Require explicit version (exclude None) tests/SeleniumTests/__init__.py38
LIST_CHROMIUM_VERSIONS	['137.0', '136.0', ...]	Chromium version pool tests/SeleniumTests/__init__.py39
LIST_FIREFOX_VERSIONS	['139.0', '138.0', ...]	Firefox version pool tests/SeleniumTests/__init__.py40
LIST_PLATFORMS	['Linux', None, 'Windows 11']	Platform pool tests/SeleniumTests/__init__.py41

Sources: tests/SeleniumTests/__init__.py35-50

Autoscaling and Parallel Execution

Variable	Default	Purpose
TEST_PARALLEL_HARDENING	false	Enable high-parallelism testing tests/SeleniumTests/__init__.py26
TEST_PARALLEL_COUNT	5	Number of test repetitions per browser tests/SeleniumTests/__init__.py27

Sources: tests/SeleniumTests/__init__.py26-27

Container Orchestration Settings (test.py)

Variable	Default	Purpose
NAMESPACE	(required)	Docker image namespace tests/test.py43
VERSION	(required)	Docker image version tag tests/test.py44
USE_RANDOM_USER_ID	false	Test with random UID (2000-65000) tests/test.py45
RUN_IN_DOCKER_COMPOSE	false	Skip container management tests/test.py46
SKIP_BUILD	false	Skip image building tests/test.py50
PLATFORMS	linux/amd64	Build platforms (comma-separated) tests/test.py51
FILESYSTEM_READ_ONLY	false	Mount root filesystem as read-only tests/test.py52
BASE_VERSION	(optional)	Base image version tests/test.py53
BASE_RELEASE	(optional)	Base image release tests/test.py54

Sources: tests/test.py43-54

Node Relay Testing

Variable	Default	Purpose
TEST_NODE_RELAY	false	Enable relay node testing tests/SeleniumTests/__init__.py29
ANDROID_PLATFORM_API	(required for Android)	Android platform API version tests/SeleniumTests/__init__.py30
TEST_PLATFORMS	linux/amd64	Platform filter for test generation tests/SeleniumTests/__init__.py31

When TEST_NODE_RELAY=Android, the framework waits 90 seconds for emulator startup tests/SeleniumTests/__init__.py61-62

Sources: tests/SeleniumTests/__init__.py29-32 tests/SeleniumTests/__init__.py61-62

---

Complete Test Execution Flow

The following diagram shows the complete end-to-end test execution flow:

Sources: tests/bootstrap.sh1-41 tests/test.py241-318 tests/SmokeTests/__init__.py26-75 tests/SeleniumTests/__init__.py145-203

---

Adding New Tests

To extend the test framework with new test cases or test classes:

Adding Test Methods to Existing Classes

Add new methods to SeleniumGenericTests for cross-browser tests tests/SeleniumTests/__init__.py65-143:

For browser-specific tests, add methods to ChromeTests, FirefoxTests, or EdgeTests tests/SeleniumTests/__init__.py145-324

Adding New Test Classes

1. Create test class inheriting from SeleniumGenericTests:

2. Update IMAGE_NAME_MAP in test.py tests/test.py61-82:

3. Update TEST_NAME_MAP in test.py tests/test.py84-107:

Adding New Test Suites

For specialized test suites (e.g., performance tests, security tests):

1. Create new module (e.g., tests/PerformanceTests/__init__.py)
2. Define test class:

3. Update test.py execution logic tests/test.py288-310:

Testing Matrix Variations

To test against multiple browser versions or platforms:

1. Enable matrix testing via environment variables tests/SeleniumTests/__init__.py35-41:

   • Set TEST_MULTIPLE_VERSIONS=true
   • Update LIST_CHROMIUM_VERSIONS or LIST_FIREFOX_VERSIONS

2. The framework automatically randomizes selection in setUp() methods tests/SeleniumTests/__init__.py161-165 tests/SeleniumTests/__init__.py275-279

Best Practices

• Use WebDriverWait: Always wait for elements rather than using time.sleep() tests/SeleniumTests/__init__.py69-70
• Inherit from SeleniumGenericTests: Reuse common test methods across browsers
• Handle exceptions gracefully: Use try-except in setUp()/tearDown() tests/SeleniumTests/__init__.py199-202
• Print timing information: Log WebDriver initialization time tests/SeleniumTests/__init__.py196-198
• Test cleanup: Ensure tearDown() quits the driver even on failure tests/SeleniumTests/__init__.py131-142

Sources: tests/SeleniumTests/__init__.py65-400 tests/test.py61-107 tests/test.py288-310