Sometimes when we try to find a WebElement from the HTML code by using the find method of selenium, we might see a NoSuchElementException or Unable to locate element error on the terminal. But it is crystal clear from the HTML code that the element is present on that page. So, why does that happen? Today we will try to figure out one of the reasons behind that.
Why Wait Functions?
I tried to run the following code in my browser but several times came up with NoSuchElementException error. It may work out in your browser properly without any additional changes. But in my case, WebDriver was unable to click the “I Accept” button of the cookie policy hence failed to proceed further.
from selenium import webdriver
driver = webdriver.Chrome(executable_path = r'G:/chromedriver_win32/chromedriver.exe')
driver.maximize_window()
driver.get('https://www.nba.com/players')
cookies = driver.find_element_by_id("onetrust-accept-btn-handler")
cookies.click()
When selenium is executing the script and clicking the button one after another, the browser requires a certain amount of time to process the request. But the script written with Python selenium does not know how much time it will take to find the button or sometimes it might not be able to find the desired button in a certain amount of time.
At that time, it will throw some kinds of script error or unexpected error because of the element that was not found at that moment or it was found after some time. In this situation, we need to wait for a certain amount of time allowing the browser to process our request properly and to execute the next line of code exactly when the earlier process is finished.
This happens because most of the web apps of recent age are built with Ajax. Ajax helps the browser to load more staff gradually. It keeps adding the elements even after the web driver finds the main page thus causing the time difference.
Python-Specific Wait or Hard Wait time.sleep()
To avoid the problem we can use python specific time.sleep() function is also called hard wait where irrespective of the case the browser is loaded or not it will wait a specific amount of time. We just need to insert the time as an argument in the time.sleep() function. We need to import the Python time module for that. So, the code will look like this:
from selenium import webdriver
import time
driver = webdriver.Chrome(executable_path = r'G:/chromedriver_win32/chromedriver.exe')
driver.maximize_window()
driver.get('https://www.nba.com/players')
time.sleep(10)
cookies = driver.find_element_by_id("onetrust-accept-btn-handler")
cookies.click()
Hopefully, the error will be fixed but the browser will wait exactly 10 seconds no matter how quickly it finds the element.
Implicit Wait
Sometimes your application might take 10 seconds to identify the element but you have been waiting for 6 seconds, in that case, you will not get the proper output in the terminal.
In the reverse case, you may set your hard wait to 20 seconds but you will get the response in 3 seconds. In that scenario even though your script finds the element in 3 seconds you have to wait for 20 seconds, the time you set as a hard wait. So, if we use this hard wait several times in a script, we will be delaying the execution and losing a great amount of time which is not a good practice. That is why selenium comes up with two features called implicit wait and explicit wait.
According to the documentation, In case of implicit wait when we are running the script the selenium will wait for the exact amount of time it takes to find the element. At the time the element is found, it will start executing the code without further delay.
from selenium import webdriver
driver = webdriver.Chrome(executable_path = r'G:/chromedriver_win32/chromedriver.exe')
driver.maximize_window()
driver.get('https://www.nba.com/players')
driver.implicitly_wait(20)
cookies = driver.find_element_by_id("onetrust-accept-btn-handler")
cookies.click()
Here, we set the implicitly wait for 20 seconds. what happens here, the driver wants to connect to the element every half a second. It keeps trying until it finds the element. Whenever it finds the element, it goes to the next line and starts executing the next line of code. Therefore, we do not need to wait for the extra amount of time as we do in case of a hard wait. if it does not find the element within the time frame of 20 seconds only then it will throw a “NoSuchElementException”.
Explicit Wait
An explicit wait is when you define a condition on a WebDriver and wait for that certain condition to occur before proceeding to the next step of the code. In explicit wait, we are doing two things at the same time. We are setting a certain time limit and at the same time, we are waiting for the occurrence of certain conditions. To use this method we need to import WebDriverWait module from selenium.webdriver.support.ui and expected_conditions module from selenium.webdriver.support.
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome(executable_path = r'G:/chromedriver_win32/chromedriver.exe')
driver.maximize_window()
driver.get('https://www.nba.com/players')
cookies = WebDriverWait(driver,10).until(EC.presence_of_element_located(('id','onetrust-accept-btn-handler')))
cookies.click()
Here in the “cookies” variable, we set the maximum time frame of 10 seconds, and “until” is another function of WebDriverWait which takes the condition statement as a parameter. By default, WebDriverWait tries to connect to the element every 500 milliseconds and if there is no response in the allocated time, it throws a TimeoutException.
Expected Conditions
As like “presence_of_element_located”, the condition we used in the ‘cookies’ variable, there are some common conditons that can be useful to automate the browser. Selenium API bindings make it easier with some available methods so that we do not need to to code an expected_condition class repeatedly.
title_istitle_containspresence_of_element_locatedvisibility_of_element_locatedvisibility_ofpresence_of_all_elements_locatedtext_to_be_present_in_elementtext_to_be_present_in_element_valueframe_to_be_available_and_switch_to_itinvisibility_of_element_locatedelement_to_be_clickablestaleness_ofelement_to_be_selectedelement_located_to_be_selectedelement_selection_state_to_beelement_located_selection_state_to_bealert_is_present
Which One is Best?
In Ajax-based web apps where the elements are already present on the web page but not yet visible due to its time-consuming page loading process, Explicit waits with a certain condition might be the best solution. It always works well in ajax driven complex web apps.
While In the case of explicit waits you need to use expected_conditions every single time you require it, a single line of code written with implicit waits is good enough for the entire script. If you are not working with complex structured web pages then it is recommended to use implicit waits since you do not need to write the codes several times in a script.
Hope it was an enjoyable reading. Feel free to check out this link to learn more about selenium waits.
To improve your Python skills, check out our free email academy: