5 Best Ways to Convert HTML Strings to Plain Text in Python

by

💡 Problem Formulation: Developers often encounter scenarios where they need to extract text from HTML data. The challenge lies in converting HTML strings, which may include a variety of tags and attributes, into plain text. For instance, if the input is <p>Hello, World!</p>, the desired output is simply “Hello, World!”. This article explores five methods to achieve this conversion in Python, each with different strengths and use-cases.

Method 1: Using BeautifulSoup

BeautifulSoup is a Python library for parsing HTML and XML documents. It provides methods for navigating, searching, and modifying parse trees. It is particularly useful for extracting text without any markup from HTML strings. This method suits complex HTML documents as BeautifulSoup can handle various intricacies of HTML parsing seamlessly.

Here’s an example:

from bs4 import BeautifulSoup

html_data = "<p>Hello, <em>World!</em></p>"
soup = BeautifulSoup(html_data, "html.parser")
plain_text = soup.get_text()

print(plain_text)

Output:

Hello, World!

This code snippet creates a BeautifulSoup object by passing our HTML data through it and specifying "html.parser" as the parsing agent. The get_text() method strips all tags and returns the plain text content.

Method 2: Using HTMLParser Class

Python’s inbuilt HTMLParser class can be extended to create a custom parser that will strip HTML tags. This method is a part of Python’s standard library, which makes it very convenient to use without extra dependencies. It is particularly good for simple HTML strings and when you want to avoid third-party packages.

Here’s an example:

from html.parser import HTMLParser

class MyHTMLParser(HTMLParser):
    def __init__(self):
        super().__init__()
        self.result = []
    
    def handle_data(self, data):
        self.result.append(data)
    
    def get_data(self):
        return ''.join(self.result)

parser = MyHTMLParser()
parser.feed("<p>Hello, <em>World!</em></p>")
print(parser.get_data())

Output:

Hello, World!

By extending the HTMLParser class, we create a custom parser that captures the character data within HTML tags using the handle_data() method. Once parsing is complete, the text can be retrieved using the get_data() method, which concatenates our stored text.

Method 3: Using re Module

The Python re (regular expression) module allows text processing through search and replace operations. It can be used to remove HTML tags and extract plain text, though it’s generally less tolerant of malformed HTML than parsers are. This method is suitable for simple HTML processing and quick scripting tasks.

Here’s an example:

import re

def strip_html_tags(html):
    clean = re.compile('<.*?>')
    return re.sub(clean, '', html)

html_data = "<p>Hello, <em>World!</em></p>"
plain_text = strip_html_tags(html_data)
print(plain_text)

Output:

Hello, World!

This code snippet showcases the use of regular expressions to remove HTML tags. The re.compile() function creates a regular expression pattern, which matches any text between (and including) < and >. The re.sub() function replaces these matched strings with an empty string, effectively stripping the HTML tags.

Method 4: Using html2text Library

html2text is a third-party Python library that converts HTML into markdown or plain text. It’s specifically built for this purpose and handles a variety of cases, such as converting links and formatting text. It is best for applications where markdown is also a beneficial output form.

Here’s an example:

import html2text

html_data = "<p>Check out my [blog](http://example.com)!</p>"
text_maker = html2text.HTML2Text()
text_maker.ignore_links = True
plain_text = text_maker.handle(html_data)

print(plain_text)

Output:

Check out my blog!

This script creates an HTML2Text object, which is then used to handle the HTML input string. We set ignore_links to True to convert the HTML without keeping markdown-style links. The handle() method produces the plain text.

Bonus One-Liner Method 5: Using lxml

lxml is a Python library for processing XML and HTML. It is very fast and can handle large amounts of data. This one-liner uses the lxml etree module to parse HTML and extract text. It works well for straightforward HTML but may not manage as well as BeautifulSoup for complex cases.

Here’s an example:

from lxml import etree

html_data = "<p>Welcome to <strong>lxml</strong> world!</p>"
plain_text = etree.HTML(html_data).xpath('//text()')

print(''.join(plain_text))

Output:

Welcome to lxml world!

The code snippet uses the etree.HTML() function to parse the HTML string and the xpath() method to select all text nodes. The result is a list of strings, which we join together to form the complete plain text.

Summary/Discussion

  • Method 1: BeautifulSoup. Strong for complex HTML. May be slower than other methods; external library.
  • Method 2: HTMLParser. Good for simple HTML. Standard library; may not handle malformed HTML well.
  • Method 3: re Module. Quick for simple tasks. Not recommended for malformed HTML; can have performance issues with large or complex HTML.
  • Method 4: html2text Library. Converts to markdown as well. External library; useful when markdown is also desired.
  • Bonus Method 5: lxml. Fast and efficient. Not as thorough as BeautifulSoup in complex situations; external library.