5 Best Ways to Calculate Word Frequency in Percentage Using Python

by

💡 Problem Formulation: When processing text with Python, you might want to determine the frequency of each word as a percentage of the total word count. This is useful for text analysis, summarization, and data preprocessing for machine learning. For example, given the input text “the quick brown fox jumps over the lazy dog”, a desired output would be to list each unique word alongside its frequency percentage.

Method 1: Using Collections and Iteration

This method employs the Collections module, which provides a high-performance container datatype. Specifically, it utilizes the Counter class to count the words and then calculates their frequency as a percentage of the total words.

Here’s an example:

from collections import Counter

def word_frequency_percentage(text):
    words = text.split()
    word_counts = Counter(words)
    total_words = len(words)
    frequencies = {word: count / total_words * 100 for word, count in word_counts.items()}
    return frequencies

# Example usage
text = "the quick brown fox jumps over the lazy dog"
result = word_frequency_percentage(text)
print(result)

Output:

{
    'the': 22.22222222222222,
    'quick': 11.11111111111111,
    'brown': 11.11111111111111,
    'fox': 11.11111111111111,
    'jumps': 11.11111111111111,
    'over': 11.11111111111111,
    'lazy': 11.11111111111111,
    'dog': 11.11111111111111
}

This code snippet takes a string of text, splits it into words, and calculates the frequency of each word using a Counter. The frequencies are then converted to percentages of the total count of words in the text.

Method 2: Using Regular Expressions and Dictionary

Regular expressions can be used to extract words from text, ensuring that punctuation is not counted as part of words. Following the extraction, a dictionary is used to calculate the frequencies and then return the percentage for each word.

Here’s an example:

import re

def word_frequency_percentage(text):
    words = re.findall(r'\w+', text)
    total_words = len(words)
    word_counts = {word: words.count(word) for word in set(words)}
    frequencies = {word: count / total_words * 100 for word, count in word_counts.items()}
    return frequencies

# Example usage
text = "the quick brown fox, jumps over the lazy dog."
result = word_frequency_percentage(text)
print(result)

Output:

{
    'the': 22.22222222222222,
    'quick': 11.11111111111111,
    'brown': 11.11111111111111,
    'jumps': 11.11111111111111,
    'over': 11.11111111111111,
    'lazy': 11.11111111111111,
    'dog': 11.11111111111111,
    'fox': 11.11111111111111
}

This code uses a regular expression to split the text into words, ensuring that words are cleanly separated from punctuation. Frequencies are then calculated similarly to Method 1, but this method may be slower due to word counting in each iteration.

Method 3: Using the pandas Library

If you are working within a data analysis context, using the pandas library can be a convenient approach to computing word frequencies. It simplifies the process of counting and converting counts to percentages for large datasets.

Here’s an example:

import pandas as pd

def word_frequency_percentage(text):
    words = pd.Series(text.split())
    frequencies = (words.value_counts() / len(words)) * 100
    return frequencies.to_dict()

# Example usage
text = "the quick brown fox jumps over the lazy dog"
result = word_frequency_percentage(text)
print(result)

Output:

{
    'the': 22.22222222222222,
    'quick': 11.11111111111111,
    'brown': 11.11111111111111,
    'fox': 11.11111111111111,
    'jumps': 11.11111111111111,
    'over': 11.11111111111111,
    'lazy': 11.11111111111111,
    'dog': 11.11111111111111
}

This snippet relies on the pandas library, which is particularly suited for handling large datasets. It uses the value_counts method to count occurrences of each unique word and then calculates their frequency as a percentage.

Method 4: Using List Comprehension and Dictionary

This approach uses list comprehension and a dictionary to achieve a more Pythonic and concise way to calculate word frequency percentages. It is straightforward and eliminates the need for external libraries.

Here’s an example:

def word_frequency_percentage(text):
    words = text.split()
    total_words = len(words)
    word_counts = {word: words.count(word) for word in set(words)}
    return {word: (count / total_words) * 100 for word, count in word_counts.items()}

# Example usage
text = "the quick brown fox jumps over the lazy dog"
result = word_frequency_percentage(text)
print(result)

Output:

{
    'the': 22.22222222222222,
    'quick': 11.11111111111111,
    'brown': 11.11111111111111,
    'fox': 11.11111111111111,
    'jumps': 11.11111111111111,
    'over': 11.11111111111111,
    'lazy': 11.11111111111111,
    'dog': 11.11111111111111
}

The code uses list comprehension and dictionary manipulation to count the frequency of each unique word. It calculates percentages directly within the dictionary comprehension without the need for additional libraries or complex functions.

Bonus One-Liner Method 5: Using Counter and a Lambda Function

This one-liner utilizes the Python Counter class combined with a lambda function to count word frequencies and convert them to percentages in a single, succinct expression.

Here’s an example:

from collections import Counter

text = "the quick brown fox jumps over the lazy dog"
print((lambda counts: {word: (count / sum(counts.values())) * 100 for word, count in counts.items()})(Counter(text.split())))

Output:

{
    'the': 22.22222222222222,
    'quick': 11.11111111111111,
    'brown': 11.11111111111111,
    'fox': 11.11111111111111,
    'jumps': 11.11111111111111,
    'over': 11.11111111111111,
    'lazy': 11.11111111111111,
    'dog': 11.11111111111111
}

This one-liner uses a lambda function to encapsulate the process of calculating the frequency percentages, making it a compact solution that can be executed in a single line of code.

Summary/Discussion

  • Method 1: Using Collections and Iteration. Strengths: Efficient and straightforward. Weaknesses: Requires import of the collections module.
  • Method 2: Using Regular Expressions and Dictionary. Strengths: Accounts for punctuation. Weaknesses: Can be slower due to repetitive word counting.
  • Method 3: Using the pandas Library. Strengths: Great for data analysis contexts and large datasets. Weaknesses: Requires installing and importing pandas, which may be overkill for simple tasks.
  • Method 4: Using List Comprehension and Dictionary. Strengths: Pythonic and requires no external libraries. Weaknesses: Potentially slower for larger texts due to repetitive operations.
  • Bonus Method 5: One-Liner Using Counter and Lambda. Strengths: Compact and elegant. Weaknesses: May be less readable for beginners.