In this article, we are going to discuss the solution to the following problems:

1. Extract all details of Offices from the website https:/directory.ntschools.net/#/offices
2. Instead of using a whole scrapy framework, use a typical Python script for extracting the data.
3. For each office, we need to scrape the Physical address, Postal address, Phone, Parent Division, and Email.
4. Also, to scrape the data by running the python script file instead of crawling scrapy from the terminal.
5. Create a Dataframe for the office details scraped from the website and print the output.

Extract all details of Offices

First, let’s quickly look at the website for data extraction.

Now disable the JavaScript in the web browser and reload the page.

We can see that nothing is loading in the browser. That means the entire website is running by JavaScript. So regular scraping will not get data from this website.

We can get the data using selenium, splash, or other methods to fetch the javascript-loaded data.

But before going for any of those solutions, we can go to the developer tab, under section network, and check for the network flow during the site loading. You can see a lot of activities are going under the network section.

Now we have to filter the content using the XHR tab. So we can see two items under the network section.

Select the `GetOfficeList` item then we can see more details of that item’s content.

Under the new section, we can see the tabs of Headers, Preview, Response, etc.

Select the Response tab, and then we can see the required details included in this link as a JSON object.

We can get all the required data by getting this JSON object from the particular request URL.

To get that URL, we can right-click on the GetOfficeList, select the copylinkaddress and store it in a variable named `base_url`.

To make a request, we require the request header details. Select the copy request headers option and store the data by converting it into a python dictionary.

Request Headers Contents

A Typical Python Script for Extracting the Data

Now we have everything to fetch the data from the website. Let’s start the spider for scraping the website.

(venv) $ scrapy genspider office_details domain

This will generate a spider basic template file.

(venv) $ cat office_details.py
import scrapy


class OfficeDetailsSpider(scrapy.Spider):
    name = 'office_details'
    allowed_domains = ['domain']
    start_urls = ['http://domain/']

    def parse(self, response):
        Pass

In this, we are removing the allowed_domains variable since we are not using it. We have to include the website URL (“https://directory.ntschools.net /#/offices”) in the strat_urls list.

(venv) $ cat office_details.py
import scrapy


class OfficeDetailsSpider(scrapy.Spider):
    name = 'office_details'
    start_urls = ['https://directory.ntschools.net/#/offices']

    def parse(self, response):
        pass

Now we are going to send the request for getting the JSON object. For that, we can write the code inside the parse function.

    def parse(self, response):
        base_url = 'https://directory.ntschools.net/api/Office/GetOfficeList'
        headers = {
            'Accept': 'application/json',
            'Accept-Encoding': 'gzip, deflate, br',
            'Accept-Language': 'en-GB,en-US;q=0.9,en;q=0.8',
            'Cache-Control': 'no-cache',
            'Connection': 'keep-alive',
            'Host': 'directory.ntschools.net',
            'Pragma': 'no-cache',
            'Referer': 'https://directory.ntschools.net/',
            'Sec-Fetch-Dest': 'empty',
            'Sec-Fetch-Mode': 'cors',
            'Sec-Fetch-Site': 'same-origin',
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36',
            'X-Requested-With': 'Fetch'
            }
        yield scrapy.Request(url=base_url, callback=self.parse_api, headers=headers)

So by using yield, the website containing the JSON object, that has been parsed to the function named parse_api.

So from the response received in the function parse_api, we can get the JSON object. Now we need to import the JSON module into our Python file for loading the JSON object into our python file.

import json

    def parse_api(self, response):
        datas = json.loads(response.body)
        for data in datas:
            print(data['name'])
            if data['offices'] is not []:
                for sub in data['offices']:
                    print(f'    {sub["name"]}')

This will give an output as follows:

This output is the same as shown on the website.

import json

    def parse_api(self, response):
        datas = json.loads(response.body)
        print(datas[0])

The output of the above code will be as follows:

If we look into the output, the postal and physical addresses are missing in this data. The office details page contained that information.

When we click on the `Agency Services` new page is loaded, https://directory. ntschools.net/#/offices/details/asdiv. But if we want to parse into the office details, we need to identify the code `asdiv`.

But this code is included in the previous JSON object with the key name `itSchoolCode`. So we are going to modify our script to get that details.

    def parse_api(self, response):
        datas = json.loads(response.body)
        for data in datas:  
            print(f"{data['name']} -> {data['itSchoolCode']}")

            if data['offices'] is not []:
                for sub in data['offices']:
                    print(f"    {sub['name']} -> {sub['itSchoolCode']}")

The output of this script is as follows:

Extract the Address From the Details Page

Now we need to fetch the URL details of the JSON object as we previously searched in the network tab.

When we loaded the office details page, additional items were generated in the network tab, as seen in the following picture:

GetOfficeDetails?param='asdiv'

Respose tab of GetOfficeDetails?param='asdiv'

Request URL for JSON object shown in the previous picture

Now we know the extension code for each office. Using this information, we can make another request to get the details of the new page. Another function with the name `parse_office`; is used for this.

Before going to function parse_office, some modifications to make in the function parse_api by adding metadata as a dictionary.

    def parse_api(self, response):
        url = 'https://directory.ntschools.net/api/Office/GetOfficeDetails?param='
        datas = json.loads(response.body)
        for data in datas: 
            code = data['itSchoolCode']
            parentDivision = {'parentDivision': data['parentDivision']}
            yield scrapy.Request(
                                url+code,
                                callback=self.parse_office,
                                meta=parentDivision,
                                headers=self.headers
                                )
            if data['offices'] is not []:
                for sub in data['offices']:
                    sub_code = sub['itSchoolCode']
                    parentDivision = {'parentDivision': sub['parentDivision']}
                    yield scrapy.Request(
                                        url+sub_code,
                                        callback=self.parse_office,
                                        meta=parentDivision,
                                        headers=self.headers
                                        )

Now we can collect all the details by writing a function name parse_office.

    def parse_office(self, response):
        parentDivision = response.meta['parentDivision']
        office_detail = json.loads(response.body)
        print({
            'OfficeName': office_detail['name'],
            'PhysicalAddress': office_detail['physicalAddress']['displayAddress'],
            'PostalAddress': office_detail['postalAddress']['displayAddress'],
            'TelephoneNumber': office_detail['telephoneNumber'],
            'FaxNumber': office_detail['facsimileTelephoneNumber'],
            'Email': office_detail['mail'],
            'Website': office_detail['uri'],
            'ParentDivision': parentDivision,
            })

Let’s see the total script developed until now:

import scrapy
import json


class OfficeDetailsSpider(scrapy.Spider):
    name = 'office_details'
    start_urls = ['https://directory.ntschools.net/#/offices']
    headers = {
            'Accept': 'application/json',
            'Accept-Encoding': 'gzip, deflate, br',
            'Accept-Language': 'en-GB,en-US;q=0.9,en;q=0.8',
            'Cache-Control': 'no-cache',
            'Connection': 'keep-alive',
            'Host': 'directory.ntschools.net',
            'Pragma': 'no-cache',
            'Referer': 'https://directory.ntschools.net/',
            'Sec-Fetch-Dest': 'empty',
            'Sec-Fetch-Mode': 'cors',
            'Sec-Fetch-Site': 'same-origin',
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36',
            'X-Requested-With': 'Fetch'
            }


    def parse(self, response):
        base_url = 'https://directory.ntschools.net/api/Office/GetOfficeList'
        yield scrapy.Request(url=base_url, callback=self.parse_api, headers=self.headers)


    def parse_api(self, response):
        url = 'https://directory.ntschools.net/api/Office/GetOfficeDetails?param='
        datas = json.loads(response.body)
        for data in datas: 
            code = data['itSchoolCode']
            parentDivision = {'parentDivision': data['parentDivision']}
            yield scrapy.Request(
                                url+code,
                                callback=self.parse_office,
                                meta=parentDivision,
                                headers=self.headers
                                )
            if data['offices'] is not []:
                for sub in data['offices']:
                    sub_code = sub['itSchoolCode']
                    parentDivision = {'parentDivision': sub['parentDivision']}
                    yield scrapy.Request(
                                        url+sub_code,
                                        callback=self.parse_office,
                                        meta=parentDivision,
                                        headers=self.headers
                                        )


    def parse_office(self, response):
        parentDivision = response.meta['parentDivision']
        office_detail = json.loads(response.body)
        print({
            'OfficeName': office_detail['name'],
            'PhysicalAddress': office_detail['physicalAddress']['displayAddress'],
            'PostalAddress': office_detail['postalAddress']['displayAddress'],
            'TelephoneNumber': office_detail['telephoneNumber'],
            'FaxNumber': office_detail['facsimileTelephoneNumber'],
            'Email': office_detail['mail'],
            'Website': office_detail['uri'],
            'ParentDivision': parentDivision,
            })

This script will generate an output as shown in the following picture:

It’s Great!

Now we scrape all the details from the API available on the website.

Extract Data by Running Python Script File Instead Crawling Scrapy from Terminal

Now we have to run the spider from the script itself and store the scraped data in a data frame and then print the data frame.

For that, we need to import CrawlerProcess class from scrapy.crawler:

from scrapy.crawler import CrawlerProcess

process = CrawlerProcess()
process.crawl(OfficeDetailsSpider)
process.start()

Here we are assigning an object, process, to CrawlerProcess() then, we passed our spider class name through the crawl() method of CrawlerProcess class. To initiate the crawling we used another method, start(). 

Create Dataframe for the Office Details Scraped from the Website

The next challenge is to build a data frame with the details we scrape from the website. For that, we have to create a global list variable and dictionary in the parse_office function to be stored in this global variable.

Import pandas as pd

office_details = list()
...
...
...
    def parse_office(self, response):
        parentDivision = response.meta['parentDivision']
        office_data = json.loads(response.body)
        office_details.append(
            {
                'OfficeName': office_data['name'],
                'PhysicalAddress': office_data['physicalAddress']['displayAddress'],
                'PostalAddress': office_data['postalAddress']['displayAddress'],
                'TelephoneNumber': office_data['telephoneNumber'],
                'FaxNumber': office_data['facsimileTelephoneNumber'],
                'Email': office_data['mail'],
                'Website': office_data['uri'],
                'ParentDivision': parentDivision,
            }
        )

We stored all the office details in a list and we can convert that list into a data frame by using the following code.

df_office = pd.DataFrame(office_details)
print(df_office)

So all the challenges have been completed and now we can look into the whole script code with a proper structure:

import scrapy
from scrapy.crawler import CrawlerProcess
import json
import pandas as pd

office_details = list()


class OfficeDetailsSpider(scrapy.Spider):
    name = 'office_details'
    start_urls = ['https://directory.ntschools.net/#/offices']
    headers = {
            'Accept': 'application/json',
            'Accept-Encoding': 'gzip, deflate, br',
            'Accept-Language': 'en-GB,en-US;q=0.9,en;q=0.8',
            'Cache-Control': 'no-cache',
            'Connection': 'keep-alive',
            'Host': 'directory.ntschools.net',
            'Pragma': 'no-cache',
            'Referer': 'https://directory.ntschools.net/',
            'Sec-Fetch-Dest': 'empty',
            'Sec-Fetch-Mode': 'cors',
            'Sec-Fetch-Site': 'same-origin',
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36',
            'X-Requested-With': 'Fetch'
            }


    def parse(self, response):
        base_url = 'https://directory.ntschools.net/api/Office/GetOfficeList'
        yield scrapy.Request(url=base_url, callback=self.parse_api, headers=self.headers)


    def parse_api(self, response):
        url = 'https://directory.ntschools.net/api/Office/GetOfficeDetails?param='
        datas = json.loads(response.body)
        for data in datas: 
            code = data['itSchoolCode']
            parentDivision = {'parentDivision': data['parentDivision']}
            yield scrapy.Request(
                                url+code,
                                callback=self.parse_office,
                                meta=parentDivision,
                                headers=self.headers
                                )
            if data['offices'] is not []:
                for sub in data['offices']:
                    sub_code = sub['itSchoolCode']
                    parentDivision = {'parentDivision': sub['parentDivision']}
                    yield scrapy.Request(
                                        url+sub_code,
                                        callback=self.parse_office,
                                        meta=parentDivision,
                                        headers=self.headers
                                        )


    def parse_office(self, response):
        parentDivision = response.meta['parentDivision']
        office_data = json.loads(response.body)
        office_details.append(
            {
                'OfficeName': office_data['name'],
                'PhysicalAddress': office_data['physicalAddress']['displayAddress'],
                'PostalAddress': office_data['postalAddress']['displayAddress'],
                'TelephoneNumber': office_data['telephoneNumber'],
                'FaxNumber': office_data['facsimileTelephoneNumber'],
                'Email': office_data['mail'],
                'Website': office_data['uri'],
                'ParentDivision': parentDivision,
            }
        )


def crawl():
    process = CrawlerProcess()
    process.crawl(OfficeDetailsSpider)
    process.start()


def main():
    crawl()
    df_office = pd.DataFrame(office_details)
    print(df_office)


if __name__ == '__main__':
    main()

The post Python Scrapy – Scraping Dynamic Website with API-Generated Content appeared first on Be on the Right Side of Change.

This article is solely for educational purposes.

Recommended Tutorial: Web Scraping – Is It Legal?

The tool used to extract data from a website is Scrapy, and the software system is UNIX operating system.

Virtual Environment Set Up

It is better to use a virtual environment for setting up the project. There are different methods to establish a virtual environment, and here we use the venv module of python for setting up our project environment.

$ python3 -m venv 'name_of_enviro'

Once created a virtual environment, it has to be used as a project environment, using the subsequent command.

$ source 'name_of_enviro'/bin/activate

After activating the virtual environment, the prompt can show the virtual environment name as follows (I am using `venv` as the name of the virtual environment.

(venv)$

 Scrapy Installation

 This command will install Scrapy in the virtual environment.

(venv)$ pip install scrapy

Creating a Project in Scrapy

Before starting to extract, we need to set up a new Scrapy Project using a directory name to store all scrapy codes and run.

(venv)$ scrapy startproject top250Movies

The above command creates a `top250Movies` directory with the following files and directories.

top250Movies
├── scrapy.cfg  # deploy configuration file
└── top250Movies  # project's Python module
    ├── __init__.py
    ├── items.py
    ├── middlewares.py
    ├── pipelines.py
    ├── settings.py
    └── spiders   # a directory for running the codes
        └── __init__.py

After starting a new project, always move to the project directory.

Our project directory is named `top250Movies`, so we move into that directory and start writing our codes by creating a python file inside the `spiders` directory.

The scrapy module can only run the project from within the project directory. Otherwise, it will generate an error.

(venv)$ cd top250Movies
(venv)$ top250Movies>

Start the coding

Let’s create a python file inside the directory named spiders.

(venv)$ top250Movies>touch top250Movies/spiders/firstSpider.py

So we created our project file, and now we need to import the library and build a spider. Spiders are the place where we define the custom behavior for crawling and parsing pages for a particular site or a group of sites).

import scrapy


class FirstSpider(scrapy.Spider):
    name = 'movies'
    start_urls = ['https://www.imdb.com/chart/top/']
    def parse(self, response):
        pass

The above codes will explain the structure of a spider.

The scraping goes through the following cycles:

1. Start by generating the initial Requests to crawl the first URL and specify a callback function to be called with the response downloaded from those requests.
2. In the callback function, we parse the response/URL and return item objects.
3. In callback functions, we parse the page contents, typically using Selectors (CSS Selector / Xpath Selector).
4. Finally, the items returned from the spider will persist in a database or be written to a file using Feed exports (JSON, CSV, etc.)

Here we need to follow the naming of variables and functions.

• `movies` is a string that defines the name of this spider. The Scrapy is to use the spider name to locate the specified spider, so it must be unique.
• `start_urls` contains a list of URLs on which the spider will begin to crawl.
• `parse(response)` is the default callback used by Scrapy to process the response received when their requests are not specifying a callback. Then, the `parse` method will process the response and return the extracted data.

Before proceeding, we can look into the scrapy shell, which is very useful to identify the items we need to scrape and execute the same to ensure that we are getting the exact result as expected.

For starting the scrapy shell, the following command will be used:

(venv)[top250Movies]$ scrapy shell
>>> 
>>>fetch('https://www.imdb.com/chart/top/')

The fetch command followed with URL will download the given URL using the Scrapy downloader and writes the contents to the response object.

The response is an object that represents an HTTP response, which is usually downloaded and fed to the Spiders for processing.

Response Parameters:

• url (string) – it will return the URL of the response.
• status (integer) – it will return the HTTP status. If the output is 200, then it is good to go.
• headers (dictionary) – it will return the headers. The dictionary values can be strings for single-valued headers and lists for multi-valued headers.
• body (bytes) – it will return the response body. To access the decoded text as a string, we use response.text.
• view (response) — it will open the response URL in a browser. Sometimes spiders see the URL pages differently from regular users. Hence this can be used to confirm what the spider sees and what we expect.

Selecting Element Attributes

There are different ways to get a value of an attribute. Here we use simple CSS syntax:

>>> response.css("td.titleColumn a::text").get()
'The Shawshank Redemption'

While inspecting the IMDb site for getting the Movie name, the particular CSS selector will be:

<td class="titleColumn">
<a href="/title/tt0111161/" title="Frank Darabont (dir.), Tim Robbins, Morgan Freeman" >The Shawshank Redemption</a>
 	<span class="secondaryInfo">(1994)</span>
</td>

For getting all movie lists, instead of the get() method, we will use the getall() method, which will return all the Movie names as a list.

Similarly, we can use the following CSS selector for getting the movie release years which is inside the <span>

>>> response.css("td.titleColumn span::text").getall()

In each code, we used `::text` after the <a> and <span> tags which will extract the text content under the each tags.

Now we are good at writing code in our spider for crawling the website.

import scrapy
class FirstSpider(scrapy.Spider):
    name = 'movies'
    start_urls = ['https://www.imdb.com/chart/top/']
  	
def parse(self, response):
    movieContent = response.css("td.titleColumn")
    for item in movieContent:
        movieName = item.css("a::text").get()
        movieYear = item.css("span::text").get()
        movieDict = {'Movie Name': movieName,
                     'Release Year': movieYear}
        yield movieDict
    pass

Here we used an additional variable `movieContent`, which stores the details of both <a> and <span> tags as list of selectors.

We can now iterate over all the titleColumn elements and put them together into a Python dictionary. Scrapy spider generates dictionaries of data extracted from the page. Hence we use the yield keyword of Python in the callback as shown in the above code.

To execute the spider without any error, we have to reach the project’s top-level directory and run:

(venv)[~/top250Movies] $ scrapy crawl movies

This command will execute the spider with ‘movies’ as names, which we have added to the spider, which will send some requests for the URL domain for getting an output similar to this:

We have extracted 250 top movie names with the released year and logged the same into our screen.

Now we will see how to store this output into a file.

To store the output data into a file, we will use the parameter -o along with the filename (JSON/CSV).

(venv)[~/top250Movies] $ scrapy crawl movies -o movieList.csv

It will create a CSV file with the name `movieList` in our project directory.

Summary

This article taught us how to install scrapy into a virtual environment.

We learned how to start a project in scrapy and the basic structure of a scrapy project folder.

We learned about the scrapy shell and the commands for getting the details from a URL

Afterward, we learned how to write the spider for scraping the IMDb website using CSS selectors.

Last but not least, we learned how to generate the output and how we can store the data into a JSON/CSV file.

Recommended Tutorial: Python Developer — Income and Opportunity

The post How to Scrape the Details of 250 Top Rated Movies in Python appeared first on Be on the Right Side of Change.