5 Best Ways to Run Selenium WebDriver with a Proxy in Python

Rate this post

πŸ’‘ Problem Formulation: When automating web browsers with Selenium WebDriver in Python, there’s often a need to route traffic through a proxy. This could be for security reasons, to bypass geo-restrictions, or to test an application with different IP addresses. This article presents solutions for setting up a proxy in Selenium WebDriver. For example, one might input proxy settings (host and port) and the expected output is a Selenium session that routes through the specified proxy server.

Method 1: Using Desired Capabilities

This method involves configuring a dictionary of desired capabilities that the Selenium WebDriver can use to route traffic through a proxy server. The capabilities include the proxy settings, which are then passed to the WebDriver when initializing the driver instance. It’s an explicit way to declare proxy preferences and can be customized for different types of proxy servers.

Here’s an example:

from selenium import webdriver

proxy = "1.2.3.4:8080"
webdriver.DesiredCapabilities.FIREFOX['proxy'] = {
    "httpProxy": proxy,
    "ftpProxy": proxy,
    "sslProxy": proxy,
    "proxyType": "MANUAL",
}

driver = webdriver.Firefox()

This would route all the browser traffic through the proxy server at 1.2.3.4 on port 8080.

This snippet sets the proxy for a Firefox WebDriver session. Our proxies for HTTP, FTP, and SSL are configured to the same server and port. We specify ‘MANUAL’ proxy type to manually configure settings.

Method 2: Using Proxy Class

The Selenium WebDriver offers a Proxy class in its ‘webdriver.common.proxy’ module. This class provides a cleaner way to set up a proxy by instantiating the Proxy object and setting its attributes directly before passing it into the desired capabilities when initializing the driver instance.

Here’s an example:

from selenium import webdriver
from selenium.webdriver.common.proxy import Proxy, ProxyType

my_proxy = "1.2.3.4:8080"
proxy = Proxy({
    'proxyType': ProxyType.MANUAL,
    'httpProxy': my_proxy,
    'sslProxy': my_proxy
})

capabilities = webdriver.DesiredCapabilities.FIREFOX
proxy.add_to_capabilities(capabilities)

driver = webdriver.Firefox(desired_capabilities=capabilities)

The browser session will use the specified proxy settings.

We set up the proxy configuration using the Selenium Proxy class and add it to Firefox’s desired capabilities. This modular approach allows for more readable code and easier proxy management.

Method 3: Configuring WebDriver Options

WebDriver options provide another approach to set a proxy. The options object specific to each browser type (e.g., ChromeOptions for Chrome) lets you add arguments or experimental options which include proxy settings, then initialize the WebDriver with these options.

Here’s an example:

from selenium import webdriver

chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--proxy-server=1.2.3.4:8080')

driver = webdriver.Chrome(options=chrome_options)

This code will open a Chrome session that utilizes the specified proxy server.

This approach uses ChromeOptions to set the proxy server for a Chrome browser session. We add the proxy argument directly, which may be more straightforward for some users, although it is less detailed than the Proxy class method.

Method 4: Using Environment Variables

By setting environment variables for the proxy before running a Selenium WebDriver script, the browser can pick up on these settings automatically. This requires no changes in the code but does affect all traffic from the machine, which might not be desirable in some cases.

Here’s an example:

import os
from selenium import webdriver

os.environ['http_proxy'] = "http://1.2.3.4:8080"
os.environ['https_proxy'] = "http://1.2.3.4:8080"
driver = webdriver.Firefox()

This will set the system’s http and https proxies, which the browser instance inherits.

Environment variables β€˜http_proxy’ and β€˜https_proxy’ are being set before the Selenium WebDriver session starts. This affects all requests by the WebDriver, as well as potentially other applications on the system during the session.

Bonus One-Liner Method 5: Using WebDriver’s –proxy-server Flag

For a quick and simple setup with Chrome or Chromium-based browsers, you can use the ‘–proxy-server’ flag when initializing the WebDriver, which doesn’t require the setup of a Proxy object or additional classes.

Here’s an example:

from selenium import webdriver

driver = webdriver.Chrome(executable_path='path_to_chromedriver', options=webdriver.ChromeOptions().add_argument("--proxy-server=1.2.3.4:8080"))

This creates a Chrome browser instance that uses the given proxy server.

This method is concise as it directly adds the proxy setting as a command-line argument when creating a new Chrome WebDriver instance, which can be useful for quick tasks or testing.

Summary/Discussion

  • Method 1: Using Desired Capabilities. Offers explicit control and detailed setup. Less readable for simple proxy configurations.
  • Method 2: Using Proxy Class. Provides clean, object-oriented setup. Requires familiarity with Selenium’s class structure.
  • Method 3: Configuring WebDriver Options. Simple and straightforward. Tightly coupled to specific browser implementations.
  • Method 4: Using Environment Variables. Affects the whole system. Useful for global proxy settings without code changes.
  • Bonus One-Liner Method 5: Using WebDriver’s –proxy-server Flag. Quick and one-liner. Limited to Chrome/Chromium browsers and less granular control.