Python Requests Library – Exception Handling & Advanced request.get() Parameters

This is the first part of a 3-part series on the Python request library:

Syntax

requests.nameofmethod(parameters)

Background & Preparation

The Requests library has several methods for GET. Part 1 and Part 2 touched on many of those available in requests.get(). This article focuses on the remaining for GET as well as Error Handling.

Add the following code to the top of each code snippet. This snippet will allow the code in this article to run error-free.

import requests

The “get” Request: “headers”

This method is not required. By default, this value is None. If True, a dictionary of HTTPS headers transfers to the specified URL.

When an HTTP request initiates, a User-Agent string transfers along with the request. This string contains the following details of your system:

The application type.
The operating system.
The software vendor.
The software version of the requesting User-Agent.

That server uses these details to determine the capability of your computer.

This code will send its header information to a server for this example.

hdrs = {
    "Connection": "keep-alive",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, 
    like Gecko) Chrome/72.0.3626.121 Safari/537.36"}

response = requests.get('https://app.finxter.com', headers=hdrs)
print(response.headers)
response.close()

Line [1] saves a well-formed User-Agent string to the hdrs variable.
Line [2] attempts to connect to the URL and sets headers to hdrs.
Line [3] outputs the header response to the terminal.
Line [4] closes the open connection

Output

{'Server': 'nginx/1.14.0 (Ubuntu)', 'Date': 'Fri, 05 Nov 2021 16:59:19 GMT', 'Content-Type': 'text/html; charset=utf-8',
'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'X-Frame-Options': 'DENY', 'Vary': 'Cookie', 'X-Content-Type-Options': 'nosniff', 'Set-Cookie': 'sessionid=0fb6y6y5d8xoxacstf74ppvacpmt2tin; expires=Fri, 19 Nov 2021 16:59:19 GMT; HttpOnly; Max-Age=19600; Path=/; SameSite=Lax', 'Content-Encoding': 'gzip'}

💡Note: This is a great Python feature. If you are interested in Web Scraping, you may want to delve further into this topic.

The “get” Request: “proxies”

If you are an avid WebScraper or need to keep your online presence hidden, using proxies is the answer. Proxies hide your IP address from the outside world.

There are several free/paid proxy services where a list of IP addresses is available and updated daily.

💡Note: The Finxter Academy does not guarantee any IP addresses. You will need to source your own.

For this example, we get a new IP address from a free proxy service and add it to a dictionary.

the_url  = 'https://somewebsite.com'
my_proxy = {"https": "https:157.245.222.225:3128"}
response = requests.get(the_url, proxies=my_proxy)
print(response.status_code)
response.close()

Line [1] set a URL to the_url variable.
Line [2] adds one fresh proxy as of this writing in the form of a dictionary.
Line [3] attempts to connect to the URL and sets proxies to my_proxy.
Line [4] outputs the status code response to the terminal.
Line [5] closes the open connection.

Output

The “get” Request: “stream”

This method is not required. By default, this value is False. If False, a response transfers indicating that the file should download immediately. If True, stream the file.

response = requests.get('https://app.finxter.com/static/favicon_coffee.png', stream=True)
print(response.status_code)
response.close()

Line [1] set the URL to the logo location and set the stream to True.
Line [2] outputs the status code response to the terminal.
Line [3] closes the open connection.

Output

Exception Handling

There is a large number of exceptions associated with the requests library. To view a detailed list, click here.

There are two ways to approach handling this situation:

Individually

For this example, we have added a timeout to requests.get(). If the connection or server times out, an exception will occur.

try:
   response = requests.get('https://app.finxter.com', timeout=(2,4))
   print(response.status_code)
   response.close()
except requests.ConnectTimeout():
    print('Timed Out!')

Line [1] initializes the try statement. The code inside here will run first.
- Line [2] attempts to connect to the URL and sets a timeout.
- Line [3] outputs the status code to the terminal.
- Line [4] closes the open connection.
Line [5] is the except statement. If a timeout occurs, the code falls to here.
- Line [6] outputs the message Timed Out! to the terminal. The script terminates.

Output

All Exceptions

All exceptions from the requests library inherit from requests.exceptions.RequestException. For this example, this code captures all exceptions.

try:
   response = requests.get('https://app.finxter.com', timeout=(2,4))
   print(response.status_code)
   response.close()
except requests.exceptions.RequestException as e:
    print(e)

Line [1] initializes the try statement. The code inside here will run first.
- Line [2] attempts to connect to the URL and sets a timeout.
- Line [3] outputs the status code to the terminal.
- Line [4] closes the open connection.
Line [5] is the except statement. If any exception occurs, the code falls here.
- Line [6] outputs the exception message (e) to the terminal. The script terminates.

Output

Bonus

You could also convert the above into a reusable function. Modify this code to meet your requirements.

def error_code(url):
    try:
        response = requests.get('https://app.finxter.c', timeout=(2,4))
    except requests.exceptions.RequestException as e:
        return e
       
nok = error_code('https://app.finxter.com')
print(nok)

Summary

In this article, we learned how to:

Use headers
Use proxies
Use a stream
Implement Exception Handling