This guide explains how to set up SeleniumBase with authenticated proxies to bypass restrictions and enhance your web scraping success.
Selenium doesn’t support authenticated proxies. Now that SeleniumWire has been deprecated for over a year already and hasn't had any updates in over two years, the solution is to use SeleniumBase. The latter project was originally designed as a wrapper for running Selenium instances to perform web automation & testing with Python. However, for us, it allows running Selenium using an authenticated proxy.
Start with installing SeleniumBase:
pip install seleniumbaseNow write a test case that will control Selenium and run the webdriver instance. The code below makes a request to the IPinfo API. Once the script receives the JSON response, it parses the response and prints its contents to the console.
from seleniumbase import BaseCase from selenium.webdriver.common.by import By import json class ProxyTest(BaseCase): def test_proxy(self): #go to the site self.driver.get("https://ipinfo.io/json") #load the json response location_info = json.loads(self.driver.find_element(By.TAG_NAME, "body").text) #iterate through the dict and print its contents for k,v in location_info.items(): print(f"{k}: {v}")Let's run the test with pytest rather than with python.
To run the test script without a proxy, use the command below:
pytest proxy_test.py -sThe response will be similar to this one:
=================================================== test session starts =================================================== platform linux -- Python 3.10.12, pytest-8.3.4, pluggy-1.5.0 rootdir: /home/nultinator/clients/bright-data/seleniumbase-proxies plugins: html-4.0.2, metadata-3.1.1, anyio-4.0.0, seleniumbase-4.34.6, ordering-0.6, rerunfailures-15.0, xdist-3.6.1 collected 1 item proxy_test.py ip: 23.28.108.255 hostname: d28-23-255-108.dim.wideopenwest.com city: Westland region: Michigan country: US loc: 42.3242,-83.4002 org: AS12083 WideOpenWest Finance LLC postal: 48185 timezone: America/Detroit readme: https://ipinfo.io/missingauth . ==================================================== 1 passed in 1.01s ==================================================== Use the --proxy flag followed by the proxy URL. Here is the template:
--proxy=your_proxy_url:your_proxy_port The following example uses a free proxy from Bright Data. The IP address is 155.54.239.64, and we’re talking to it on port 80.
--proxy=155.54.239.64:80 Authenticated proxies are handled the same way. Simply include your username and password in the URL:
proxy=<YOUR_USERNAME>:<YOUR_PASSWORD>@<PROXY_URL>:<PROXY_PORT> The best authenticated proxies options are:
- Residential proxies: use real user IPs which makes them ideal for bypassing bot detection.
- Datacenter proxies: faster and more cost-effective but easier to detect.
- ISP proxies: combine the benefits of both, offering speed with high trust levels.
The example below is set up to run with one of our proxies at Bright Data. Be sure to replace the username, zone name, and password with your own credentials.
pytest proxy_test.py --proxy=brd-customer-<YOUR-USERNAME>-zone-<YOUR-ZONE-NAME>:<YOUR-PASSWORD>@brd.superproxy.io:33335 -sRunning it results in the following output:
=================================================== test session starts =================================================== platform linux -- Python 3.10.12, pytest-8.3.4, pluggy-1.5.0 rootdir: /home/nultinator/clients/bright-data/seleniumbase-proxies plugins: html-4.0.2, metadata-3.1.1, anyio-4.0.0, seleniumbase-4.34.6, ordering-0.6, rerunfailures-15.0, xdist-3.6.1 collected 1 item proxy_test.py ip: 144.202.4.246 hostname: 144-202-4-246.lum-int.io city: Piscataway region: New Jersey country: US loc: 40.4993,-74.3990 org: AS20473 The Constant Company, LLC postal: 08854 timezone: America/New_York readme: https://ipinfo.io/missingauth . ==================================================== 1 passed in 3.25s ==================================================== With Bright Data's proxies, you can choose your location by using the country flag and a two-letter country code:
pytest proxy_test.py --proxy=brd-customer-<YOUR-USERNAME>-zone-<YOUR-ZONE-NAME>:<YOUR-PASSWORD>-country-es@brd.superproxy.io:33335 -sWhen you use es (Spain) as your country code, you get routed through a proxy in Spain. You can verify this in the output below.
=================================================== test session starts =================================================== platform linux -- Python 3.10.12, pytest-8.3.4, pluggy-1.5.0 rootdir: /home/nultinator/clients/bright-data/seleniumbase-proxies plugins: html-4.0.2, metadata-3.1.1, anyio-4.0.0, seleniumbase-4.34.6, ordering-0.6, rerunfailures-15.0, xdist-3.6.1 collected 1 item proxy_test.py ip: 176.119.14.158 city: Paracuellos de Jarama region: Madrid country: ES loc: 40.5035,-3.5278 org: AS203020 HostRoyale Technologies Pvt Ltd postal: 28860 timezone: Europe/Madrid readme: https://ipinfo.io/missingauth . ==================================================== 1 passed in 3.98s ==================================================== Please refer to the geolocation documentation for details.
The code below uses a set of country codes, but these can easily be replaced with actual proxy IPs. countries holds the list of country codes. Next, we iterate through them and run our proxy test with all four country codes.
us: United Stateses: Spainil: Israelgb: Great Britain
import subprocess #list of country codes countries = [ "us", "es", "il", "gb", ] #iterate through the countries and make a shell command for each one for country in countries: command = f"pytest proxy_test.py --proxy=brd-customer-<YOUR-USERNAME>-<YOUR-ZONE-NAME>-country-{country}:[email protected]:33335 -s" #run the shell command subprocess.run(command, shell=True)You can run this as a regular Python file:
python rotate_proxies.pyThe output should be similar to this:
(Linux uses --headless by default. To override, use --headed / --gui. For Xvfb mode instead, use --xvfb. Or you can hide this info by using --headless / --headless2 / --uc.) =================================================== test session starts =================================================== platform linux -- Python 3.10.12, pytest-8.3.4, pluggy-1.5.0 rootdir: /home/nultinator/clients/bright-data/seleniumbase-proxies plugins: html-4.0.2, metadata-3.1.1, anyio-4.0.0, seleniumbase-4.34.6, ordering-0.6, rerunfailures-15.0, xdist-3.6.1 collected 1 item proxy_test.py ip: 164.90.142.33 city: Clifton region: New Jersey country: US loc: 40.8344,-74.1377 org: AS14061 DigitalOcean, LLC postal: 07014 timezone: America/New_York readme: https://ipinfo.io/missingauth . ==================================================== 1 passed in 3.84s ==================================================== (Linux uses --headless by default. To override, use --headed / --gui. For Xvfb mode instead, use --xvfb. Or you can hide this info by using --headless / --headless2 / --uc.) =================================================== test session starts =================================================== platform linux -- Python 3.10.12, pytest-8.3.4, pluggy-1.5.0 rootdir: /home/nultinator/clients/bright-data/seleniumbase-proxies plugins: html-4.0.2, metadata-3.1.1, anyio-4.0.0, seleniumbase-4.34.6, ordering-0.6, rerunfailures-15.0, xdist-3.6.1 collected 1 item proxy_test.py ip: 5.180.9.15 city: Madrid region: Madrid country: ES loc: 40.4066,-3.6724 org: AS203020 HostRoyale Technologies Pvt Ltd postal: 28007 timezone: Europe/Madrid readme: https://ipinfo.io/missingauth . ==================================================== 1 passed in 3.60s ==================================================== (Linux uses --headless by default. To override, use --headed / --gui. For Xvfb mode instead, use --xvfb. Or you can hide this info by using --headless / --headless2 / --uc.) =================================================== test session starts =================================================== platform linux -- Python 3.10.12, pytest-8.3.4, pluggy-1.5.0 rootdir: /home/nultinator/clients/bright-data/seleniumbase-proxies plugins: html-4.0.2, metadata-3.1.1, anyio-4.0.0, seleniumbase-4.34.6, ordering-0.6, rerunfailures-15.0, xdist-3.6.1 collected 1 item proxy_test.py ip: 64.79.233.151 city: Tel Aviv region: Tel Aviv country: IL loc: 32.0809,34.7806 org: AS9009 M247 Europe SRL timezone: Asia/Jerusalem readme: https://ipinfo.io/missingauth . ==================================================== 1 passed in 3.36s ==================================================== (Linux uses --headless by default. To override, use --headed / --gui. For Xvfb mode instead, use --xvfb. Or you can hide this info by using --headless / --headless2 / --uc.) =================================================== test session starts =================================================== platform linux -- Python 3.10.12, pytest-8.3.4, pluggy-1.5.0 rootdir: /home/nultinator/clients/bright-data/seleniumbase-proxies plugins: html-4.0.2, metadata-3.1.1, anyio-4.0.0, seleniumbase-4.34.6, ordering-0.6, rerunfailures-15.0, xdist-3.6.1 collected 1 item proxy_test.py ip: 185.37.3.107 city: London region: England country: GB loc: 51.5085,-0.1257 org: AS9009 M247 Europe SRL postal: E1W timezone: Europe/London readme: https://ipinfo.io/missingauth . ==================================================== 1 passed in 2.90s ==================================================== SeleniumBase opens up many web scraping capabilities in Selenium. Unlock the full potential of Selenium-based scraping with Bright Data’s industry-leading proxy services. Start your free trial today!