Python Requests Library – Understanding requests.get()

Rate this post

This is the first part of a 3-part series on the Python request library:

  1. Python Requests Library – Your First HTTP Request in Python
  2. Python Requests Library – Understanding requests.get() Parameters
  3. Python Requests Library – Exception Handling & Advanced request.get() Parameters

Syntax

requests.nameofmethod(parameters)

Background and Preparation

The Requests library has several options for GET. Part 1 touched on the requests.get() URL and timeout. This article focuses on additional methods for GET.

If the Requests library does not exist on the computer, navigate to Part 1 for instructions.


The “get” Request: “params”

At some point, you may need to send data using a URL query string. If the query is hard-coded, the format would be similar to the below.

Example: https://somewebsite.com?key1=val&key2=val

💡 Note: The first argument contains a question mark (?) to signify a single value. If passing more than one, use the ampersand (&) between additional values.

The requests library allows you to easily pass these arguments as one of the following data types:

For this example, the test website httpbin is the URL.

import requests
key_vals  = {'key1': 'value1', 'key2': 'value2'}
response = requests.get('https://httpbin.org/get', params=key_vals)
print(response.url)
response.close()
  • Line [1] imports the requests library.
  • Line [2] assigns two key:value pairs to a dictionary.
  • Line [3] attempts to connect to the URL and the key_vals dictionary to params.
  • Line [4] outputs the URL with the contents of key_vals appended.
  • Line [5] closes the open connection.

Output

https://httpbin.org/get?key1=value1&key2=value2

The “get” Request: “allow_redirects”

This method is not required and can be True or False. By default, this value is True: allowing redirects. If False, the code prevents redirection to another website or another web page on the same site.

import requests
response = requests.get('https://app.finxter.com', allow_redirects=False)
print(response.status_code)
response.close()
  • Line [1] imports the requests library.
  • Line [2] attempts to connect to the URL and sets allow_redirects to False.
  • Line [3] outputs the response code to the terminal.
  • Line [4] closes the open connection.

Output

302

The “get” Request: “auth”

Often referred to as Basic Authentication, this is one of the simplest methods to use. This option is not required. By default, this value is None: no authentication required. The format is a tuple with two elements.

import requests
response = requests.get('https://www.facebook.com/', auth=('username', 'password'))
print(response.status_code)
response.close()
  • Line [1] imports the requests library.
  • Line [2] attempts to connect to the and sets auth to a username and password.
  • Line [3] outputs the response code to the terminal.
  • Line [4] closes the open connection.

Output

200

For additional authentication methods, click here.


The “get” Request: “cert” and “verify”

This method requires a valid SSL certificate. This certificate is used for HTTPS requests.

The SSL certificate is a small file that connects the specified certificate to a company’s details. A website with an SSL certificate is assumed to be secure. By default, cert equals Enabled: checks for a valid SSL certificate. If the SSL certificate is invalid, an SSLError will occur.

import requests
response = requests.get('https://somesite.com', cert='certs/my_cert.cert')
print(response.status_code)
response.close()
  • Line [1] imports the requests library.
  • Line [2] attempts to connect to the URL and sets cert to the location and filename of the SSL certificate.
  • Line [3] outputs the response code to the terminal.
  • Line [4] closes the open connection.

Output

200

If unsuccessful, an error code outputs to the terminal displaying the details. Perhaps the SSL certificate was not set up or improperly set up. To get around this, use verify and set to False.

response = requests.get('https://somesite.com', cert='certs/my_cert.cert', verify=False)
print(response.status_code)
response.close()

Output

For this example, the `successful` status code is returned. However, we did get a warning about validation.

<Response [200]>
    ...
    Unverified HTTPS request is being made to host 'somesite.com'. Adding certificate verification is strongly advised. 
    ...

The “get” Request: “cookies”

This method is not required and is a dictionary of cookies sent to a specified URL. By default, the value is None: no cookies sent.

This example uses the test website httpbin and issues a custom cookie to a URL.

my_cookies  = dict(cookies_are='working')
response = requests.get('http://httpbin.org/cookies', cookies=cookies)
print(response.text)
response.close()
  • Line [1] creates a cookie.
  • Line [2] passes a URL and sets cookies to my_cookies.
  • Line [3] outputs the contents to the terminal.
  • Line [4] closes the open connection.

Output

{
  "cookies": {
    "cookies_are": "working"
  }
}

Summary

In this article, we learned how to:

  • Send data via a URL
  • Allow or prevent redirects
  • Use authentication
  • Use an SSL certificate and verify the same
  • Use cookies