5 Best Ways to Fetch Text from Wikipedia’s Infobox in Python

by

💡 Problem Formulation: Extracting structured data from Wikipedia’s infoboxes can be a valuable task for data scientists and researchers looking to aggregate informational summaries from various topics. Given a Wikipedia page as an input, the goal is to programmatically retrieve the content of its infobox in Python, and output this as plain text, a dictionary, or other structured forms.

Method 1: Using BeautifulSoup and Requests

BeautifulSoup is a Python library used for web scraping purposes to pull the data out of HTML and XML files. It creates parse trees that is helpful to extract the data easily. When combined with Requests, another library for making HTTP requests in Python, BeautifulSoup can be utilized to fetch and parse the infobox from a Wikipedia page.

Here’s an example:

import requests
from bs4 import BeautifulSoup

def get_infobox(url):
    response = requests.get(url)
    soup = BeautifulSoup(response.content, 'html.parser')
    infobox = soup.find('table', {'class':'infobox'})
    return infobox.get_text()

url = 'https://en.wikipedia.org/wiki/Python_(programming_language)'
infobox_text = get_infobox(url)
print(infobox_text)

Output:

Python
Paradigm
Multi-paradigm: functional, imperative, object-oriented, reflective
...
Website
www.python.org

This code snippet defines a function get_infobox that takes a Wikipedia page URL as an argument, fetches the HTML content using requests.get(), and then parses it using BeautifulSoup. We search for the table with class ‘infobox’, and extract its text. Though effective, the result may need further cleaning depending on the structure of the infobox.

Method 2: Using Pywikibot

Pywikibot is a Python library and a collection of tools that automate work on MediaWiki sites. It interacts with the Wikipedia API to provide direct access to the server data, allowing retrieval and modification of information on any MediaWiki website. For extracting data from an infobox, it accesses the raw wiki text and parses it accordingly.

Here’s an example:

import pywikibot

site = pywikibot.Site('en', 'wikipedia')
page = pywikibot.Page(site, 'Python (programming language)')
templates = page.templatesWithParams()
for template in templates:
    if 'Infobox' in template[0].title():
        infobox = template
        break
print(infobox)

Output:

('Template:Infobox programming language', [...])

The code uses Pywikibot to connect to the English Wikipedia site, retrieves the page for the Python programming language, and then iterates over the templates associated with the page. By checking for ‘Infobox’ in the template title, we capture the infobox details. This approach accesses the raw data of the infobox but might be more complex to set up and use compared to scraping.

Method 3: Using Wikipedia-API

Wikipedia-API is a Python package that makes it easy to access and parse data from Wikipedia. It is built on top of the Wikipedia’s public API and allows for simple and direct querying of Wikipedia pages, sections, and infoboxes, ideal for developers looking to fetch infobox text without much overhead.

Here’s an example:

import wikipediaapi

wiki_wiki = wikipediaapi.Wikipedia('en')
page_py = wiki_wiki.page('Python (programming language)')
print(page_py.text[:60])

Output:

Python is an interpreted high-level general-purpose programming ...

This snippet utilizes the wikipediaapi library to query the English Wikipedia for the “Python (programming language)” page, and then prints the text section of the page. Note that this code does not extract the infobox as such, it only showcases how to get started with fetching page content using the Wikipedia-API.

Method 4: Using wptools

wptools is another Python library that wraps the Wikimedia API to make it easier to access and parse data from Wikipedia and other Wikimedia sites. It is designed for more in-depth data extraction, including pulling information specifically from infoboxes within Wikipedia articles.

Here’s an example:

import wptools

page = wptools.page('Python (programming language)').get_parse()
infobox = page.data['infobox']
for key, value in infobox.items():
    print("{}: {}".format(key, value))

Output:

name: Python
paradigm: [[Multi-paradigm programming language|Multi-paradigm]]: ...
website: {{URL|https://www.python.org/}}

This code uses the wptools library to fetch the parsed content of the ‘Python (programming language)’ Wikipedia page. After calling get_parse(), it directly accesses the ‘infobox’ key within the returned data structure. This method decouples the infobox data from the page text, providing a convenient way to handle infobox content.

Bonus One-Liner Method 5: Using pandas.read_html

Pandas is a powerful data analysis tool for Python which, among other things, can also read HTML table data directly into a DataFrame. In cases where the Wikipedia infobox is structured as a plain HTML table, Pandas can be used to fetch and parse this information with a one-liner.

Here’s an example:

import pandas as pd

infoboxes = pd.read_html('https://en.wikipedia.org/wiki/Python_(programming_language)', match='Infobox', index_col=0)
infobox = infoboxes[0]
print(infobox.to_dict(orient='index'))

Output:

{'Paradigm': {1: 'Multi-paradigm: functional, imperative, object-oriented...'}, ...}

Pandas’ read_html function takes a URL and optional match parameter to filter for specific tables—in this case, ‘Infobox’. It reads the matching tables into a list of DataFrames. The first DataFrame is typically the infobox table, which can then be converted to a dictionary or another structure as shown.

Summary/Discussion

  • Method 1: BeautifulSoup and Requests. Strengths: Easy to use and versatile; can handle many different HTML structures. Weaknesses: Might require additional parsing and does not leverage the Wikipedia API directly.
  • Method 2: Pywikibot. Strengths: Powerful, interacts directly with Wikipedia API. Weaknesses: Steeper learning curve, more complex setup required.
  • Method 3: Wikipedia-API. Strengths: Simple and user-friendly. Weaknesses: Might not be as detailed or flexible as other methods for accessing structured data like infoboxes.
  • Method 4: wptools. Strengths: Specifically tailored for Wikipedia data extraction, including infoboxes. Weaknesses: Less known and might have a steeper learning curve than BeautifulSoup.
  • Bonus Method 5: pandas.read_html. Strengths: One-liner, very efficient for well-structured HTML tables. Weaknesses: Depends on the infobox being in the form of a clean HTML table.