Python Selenium – How to Scroll Down a Page?

If we want to move down to a page or want to search for something on a page that is not in view, we use scrolling to reach there. Is it possible to scroll a page automatically with selenium? Selenium‘s main feature does not have an option for scrolling. But we can achieve it with some additional javascript features enabled in Python by using the driver object. In selenium, it is possible to scroll down the page in three different ways. Today we will try to know about all three possible options.

Set Up the Environment

Since I have a passion for traveling, I love to read travel blogs. Today we will try to scroll a website called “The 50 most visited tourist attractions in the world”.

So, let us start the process. First, we need to import the WebDriver from selenium and then create a driver object from it. Next, we need to specify the path of the ChromeDriver as we will be using a chrome browser to scroll the page. The maximize_window() method is available to have a better view. Then we will try to connect to the website using the driver.get() method. We will be using implicit wait for 10 seconds. A cookie policy will appear at the bottom of the page when we will be connected. We need to find the WebElement of “OK, got it” button to accept it and then click it.

from selenium import webdriver
driver = webdriver.Chrome(executable_path = r'G:/chromedriver_win32/chromedriver.exe')
driver.maximize_window()
driver.get('https://www.lovehomeswap.com/blog/latest-news/the-50-most-visited-tourist-attractions-in-the-world')
driver.implicitly_wait(10)
cookie = driver.find_element_by_link_text('OK, got it')
cookie.click()

Scroll Down the Page by Pixel

It is possible to scroll a page with pixel number. There is a method called execute_script() which enables us to scroll a page. The command goes like this:

driver.execute_script("window.scrollBy(0,500)","")

Here we need to input two parameters in the scrollBy() method. 0 is the starting pixel position or default pixel and 500 is the pixel position we want to scroll to. By changing these values, it is possible to scroll down from one place to another place. The second parameter of the execute_script() method will remain empty. Let’s try to do it on the website.

driver.execute_script("window.scrollBy(0,3000)","")

As we have set the 2nd parameter of the scrollBy() method from 500 to 3000 we can see the scroll bar at the right-hand side does not remain in its default position. It has scrolled down a bit where the 3000-pixel position lies. By changing the second parameter we can visit certain places on a page with the help of this method.

Scroll Down the Page Till the Element Found

Now we want to search for a specific element in the webpage but we don’t know the exact pixel position for that point. How can we scroll down to that specific element?  that is also possible with the following command.

driver.execute_script("arguments[0].scrollIntoView();",Element)

To work with this command, at first we need to identify the element that we want to view, and then we will store it to a variable. Again we will use execute_script() method and it will take two parameters as well. We will input “arguments[0].scrollIntoView();” as the first parameter and the variable that contains the identified element as the second parameter. Hopefully, the scroll bar will automatically move to the place where the element is located.

Let’s try to find the element "Niagra Falls" from the web page. We want to set our scroller to view this element automatically. Following code will be good enough.

niagara_falls = driver.find_element_by_link_text('Niagara Falls')
driver.execute_script("arguments[0].scrollIntoView();",niagara_falls)

Here, we tried to find the element by link text and created a variable niagara_falls with the located WebElement. In the next line, we executed the command with driver.execute_script method.

Niagara Falls appears at the top of the page. sometimes it may not be visible due to the “Log in” bar. Then you need to scroll up a little to get the view.

Scroll Down Till the End of the Page

It is also possible with the execute_script method to scroll down to the end of any page. the command will look like this:

driver.execute_script("window.scrollBy(0,document.body.scrollHeight)")

Here again, we will use javascript statement inside the execute_script() method. The scrollBy() method will take two parameters. The first one is 0 as the initial starting point and the second one should be javascript defined β€œdocument.body.scrollHeight” as it helps the scroller to reach the ending point of the page. If we run the code we will see the page like this.

We can see from the right-hand side that the scrollbar reached the ending point of the page.

So, that’s all about the methods we use nowadays in selenium python to automatically scroll down the browser. I hope you will find this link useful to learn more about Selenium.