Set Up the Environment
Since I have a passion for traveling, I love to read travel blogs. Today we will try to scroll a website called “The 50 most visited tourist attractions in the world”.
So, let us start the process. First, we need to import the
WebDriver from selenium and then create a driver object from it. Next, we need to specify the path of the
ChromeDriver as we will be using a chrome browser to scroll the page. The
maximize_window() method is available to have a better view. Then we will try to connect to the website using the
from selenium import webdriver driver = webdriver.Chrome(executable_path = r'G:/chromedriver_win32/chromedriver.exe') driver.maximize_window() driver.get('https://www.lovehomeswap.com/blog/latest-news/the-50-most-visited-tourist-attractions-in-the-world') driver.implicitly_wait(10) cookie = driver.find_element_by_link_text('OK, got it') cookie.click()
Scroll Down the Page by Pixel
It is possible to scroll a page with pixel number. There is a method called
execute_script() which enables us to scroll a page. The command goes like this:
Here we need to input two parameters in the
scrollBy() method. 0 is the starting pixel position or default pixel and 500 is the pixel position we want to scroll to. By changing these values, it is possible to scroll down from one place to another place. The second parameter of the
execute_script() method will remain empty. Let’s try to do it on the website.
As we have set the 2nd parameter of the
scrollBy() method from 500 to 3000 we can see the scroll bar at the right-hand side does not remain in its default position. It has scrolled down a bit where the 3000-pixel position lies. By changing the second parameter we can visit certain places on a page with the help of this method.
Scroll Down the Page Till the Element Found
Now we want to search for a specific element in the webpage but we don’t know the exact pixel position for that point. How can we scroll down to that specific element? that is also possible with the following command.
To work with this command, at first we need to identify the element that we want to view, and then we will store it to a variable. Again we will use
execute_script() method and it will take two parameters as well. We will input “
arguments.scrollIntoView();” as the first parameter and the variable that contains the identified element as the second parameter. Hopefully, the scroll bar will automatically move to the place where the element is located.
Let’s try to find the element
"Niagra Falls" from the web page. We want to set our scroller to view this element automatically. Following code will be good enough.
niagara_falls = driver.find_element_by_link_text('Niagara Falls') driver.execute_script("arguments.scrollIntoView();",niagara_falls)
Here, we tried to find the element by link text and created a variable
niagara_falls with the located
WebElement. In the next line, we executed the command with
Niagara Falls appears at the top of the page. sometimes it may not be visible due to the “Log in” bar. Then you need to scroll up a little to get the view.
Scroll Down Till the End of the Page
It is also possible with the
execute_script method to scroll down to the end of any page. the command will look like this:
execute_script() method. The
document.body.scrollHeight” as it helps the scroller to reach the ending point of the page. If we run the code we will see the page like this.
We can see from the right-hand side that the scrollbar reached the ending point of the page.
So, that’s all about the methods we use nowadays in selenium python to automatically scroll down the browser. I hope you will find this link useful to learn more about Selenium.