How to Send a POST Request Using urllib in Python?

by

In this article, you will learn how to send a POST request using the urllib library in Python.

Once you have gone through the procedure to send POST requests via urllib, we will also have a look at how to send POST requests using the requests library which is used to handle GET and POST requests in Python. So, get ready to unearth the answers to the following questions –

  • How to send a POST request using urllib in Python?
  • How to send a POST request using the requests library in Python?

You can use a POST request to send data to a server to create/update a resource. When you send the data to a server using a POST request, the data is stored in the request body of the HTTP request. For example, you can use a POST request to send ‘key:value’ pairs in a dictionary to a web server for storage.

Solution: Use urllib.request.urlopen() and urllib.request.Request()

Approach: Call the urlencode(data) method of the urllib.parse library to encode the data object (dictionary data) to a string. Create the request object using Request(url, urlencode(data).encode()). Here, url denotes the url of the source/server while the second parameter is the encoded data object. Once the request object has been created, send the POST request by calling urlopen(req) where req is the request object.

The response returned by the server is a file-like object. You can read the body of this response, i.e., the HTML content contained within the response using the read() function which will return the HTML content of the response as a byte object. Use the decode() function to convert this byte object to a string. All of this can be done in a single line like so: urlopen(req).read().decode()

Code:

from urllib.parse import urlencode
from urllib.request import Request, urlopen

url = 'https://httpbin.org/post' # Set destination URL here
data = {'name': 'Bruce Wayne'}     # Set POST fields here
req = Request(url, urlencode(data).encode())
response = urlopen(req).read().decode()
print(response)

Output:

{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {
    "name": "Bruce Wayne"
  }, 
  "headers": {
    "Accept-Encoding": "identity", 
    "Content-Length": "16", 
    "Content-Type": "application/x-www-form-urlencoded", 
    "Host": "httpbin.org", 
    "User-Agent": "Python-urllib/3.9", 
    "X-Amzn-Trace-Id": "Root=1-6332ab8d-4c3577a95da062ec0172ee78"
  }, 
  "json": null, 
  "origin": "49.37.37.140", 
  "url": "https://httpbin.org/post"
}

NOTE:

urllib.request.Request() represents a Request object that represents the HTTP request you want to make to the source URL. Actually, the class urllib.request.Request() is nothing but an abstraction of the URL request.

You can define the urllib.request.Request class as: class urllib.request.Request(url, data=None, headers{}, origin_req_host=None, unverifiable=False, method=None)

Now that you have conquered the art of sending POST requests via urllib, let’s dive into some other Python libraries that also allow you to send POST requests.

Sending a POST Request Using the requests Library in Python

The requests library facilitates you with the post() method to send a POST request to a web server. You can simply send the dictionary data along with the request url in your post request as: requests.post(url, dict_data)

Code:

import requests

url = 'https://httpbin.org/post'
params = {'name': 'Bruce Wayne'}
res = requests.post(url, params)
print(res.text)

Output:

{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {
    "name": "Bruce Wayne"
  }, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Content-Length": "16", 
    "Content-Type": "application/x-www-form-urlencoded", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.27.1", 
    "X-Amzn-Trace-Id": "Root=1-6332b10b-6b9deea17583b16f31c092f5"
  }, 
  "json": null, 
  "origin": "49.37.37.140", 
  "url": "https://httpbin.org/post"
}

Example 2: The following example demonstrates how you can send JSON data in your POST request. You can use the json module in this case and supply a json formatted string within the POST request. This can be done using json.dumps({“key”: “value”}).

import requests
import json

url = 'https://httpbin.org/post'
payload = {'Spiderman': 'Peter Parker'}
res = requests.post(url, data=json.dumps(payload))
print(res.text)

Output:

{
  "args": {}, 
  "data": "{\"Spiderman\": \"Peter Parker\"}", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Content-Length": "29", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.27.1", 
    "X-Amzn-Trace-Id": "Root=1-6332b266-35e371d31afd6c6d59e3b192"
  }, 
  "json": {
    "Spiderman": "Peter Parker"
  }, 
  "origin": "49.37.37.140", 
  "url": "https://httpbin.org/post"
}

Exercise

Challenge: Before we wrap our discussion here’s a task for you. Consider that you have been given the following url: “http://eprint.com.hr/demo/login.php“. Use the username as ‘User1’ and the password as ‘123456’. Can you use a POST request to successfully login?

Solution: We will be using the requests library to solve this programming challenge. Define a dictionary to store the key-value pairs: 'username':'User1' and 'password':'123456'. You can send these values as data along with the POST request to initiate the login. Here’s the code that demonstrates the solution:

Code:

import requests
url = 'http://eprint.com.hr/demo/login.php'


def submit_form():
    payload = {'username': 'User1', 'password': '123456'}
    resp = requests.post(url, payload)
    print(resp.text)


submit_form()

Conclusion

With that we come to the end of this tutorial and I hope this helped you with your queries. You learned how to send POST requests using the urllib and requests library with the help of numerous examples.

Please subscribe and stay tuned for more interesting discussions and solutions. Happy coding! 🙂

Web Scraping with BeautifulSoup

One of the most sought-after skills on Fiverr and Upwork is web scraping.

Make no mistake: extracting data programmatically from websites is a critical life skill in today’s world that’s shaped by the web and remote work.

This course teaches you the ins and outs of Python’s BeautifulSoup library for web scraping.