5 Best Ways to Tokenize Text Using NLTK in Python

by

💡 Problem Formulation: Tokenizing text is the process of breaking down a text paragraph into smaller chunks, such as words or sentences. This is a fundamental task in Natural Language Processing (NLP) that prepares text for deeper analysis and understanding. For example, the input “NLTK is awesome!” would be tokenized into the output [“NLTK”, “is”, “awesome”, “!”].

Method 1: Word Tokenization using nltk.word_tokenize()

Word tokenization is a common approach where the text is segmented into individual words. The nltk.word_tokenize() function is highly versatile and can handle complex word tokenization effortlessly. It is based on the Penn Treebank Tokenization and considers punctuation as separate tokens.

Here’s an example:

import nltk
nltk.download('punkt')

from nltk.tokenize import word_tokenize

text = "Let's tokenize this string!"
tokens = word_tokenize(text)
print(tokens)

Output:

['Let', "'s", 'tokenize', 'this', 'string', '!']

This code snippet begins by importing the NLTK package and downloading the necessary datasets. It then imports the word_tokenize method and applies it to a sample text, resulting in a list of tokens that includes punctuation and contractions as separate tokens.

Method 2: Sentence Tokenization using nltk.sent_tokenize()

Sentence tokenization involves dividing a text into its constituent sentences. This can be effectively done using the nltk.sent_tokenize() function, which is equipped to handle various sentence-ending punctuation and capitalization cues.

Here’s an example:

from nltk.tokenize import sent_tokenize

text = "Hello world. Python is great! Isn't it?"
sentences = sent_tokenize(text)
print(sentences)

Output:

['Hello world.', 'Python is great!', "Isn't it?"]

After importing the sent_tokenize method from NLTK, the example demonstrates its application to a string of text. The output is an array of sentences segmented based on typical end-of-sentence punctuation.

Method 3: Punkt Sentence Tokenizer

The Punkt sentence tokenizer is a pre-trained model used to divide a text into a list of sentence tokens. It works well even for texts containing abbreviations and other complexities.

Here’s an example:

from nltk.tokenize import PunktSentenceTokenizer

tokenizer = PunktSentenceTokenizer()
text = "Mr. Smith loves Python. But what about Mr. Anderson?"
sentences = tokenizer.tokenize(text)
print(sentences)

Output:

['Mr. Smith loves Python.', 'But what about Mr. Anderson?']

This method first involves instantiating a PunktSentenceTokenizer object. When the tokenize method of this object is called with a string of text, it produces a list of sentence tokens that correctly handle cases with periods used in abbreviations.

Method 4: Regex Tokenization using nltk.RegexpTokenizer

When specific patterns of tokenization are needed, one can use the nltk.RegexpTokenizer class. This method allows the definition of a regular expression pattern to specify what constitutes a token.

Here’s an example:

from nltk.tokenize import RegexpTokenizer

tokenizer = RegexpTokenizer(r'\w+')
text = "Tokens, created from RegEx: won't split on apostrophes."
tokens = tokenizer.tokenize(text)
print(tokens)

Output:

['Tokens', 'created', 'from', 'RegEx', 'won', 't', 'split', 'on', 'apostrophes']

In this snippet, a RegexpTokenizer instance is created with a regex pattern that captures word characters. When used to tokenize the text, it generates tokens while ignoring punctuation (except for apostrophes inside words).

Bonus One-Liner Method 5: Whitespace Tokenization with nltk.WhitespaceTokenizer

The Whitespace tokenizer simply uses whitespace to tokenize text. This method is fast and straightforward but does not regard punctuation as separate tokens.

Here’s an example:

from nltk.tokenize import WhitespaceTokenizer

tokenizer = WhitespaceTokenizer()
text = "Simple tokenization\tbased on spaces and tabs."
tokens = tokenizer.tokenize(text)
print(tokens)

Output:

['Simple', 'tokenization', 'based', 'on', 'spaces', 'and', 'tabs.']

This minimal example employs the WhitespaceTokenizer to segment a string of text at occurrences of space or tab characters. It provides a basic token list without any consideration for punctuation or contractions.

Summary/Discussion

  • Method 1: Word Tokenization with word_tokenize(). Strengths: Handles punctuation and contractions effectively. Weaknesses: May not be ideal for custom tokenization patterns.
  • Method 2: Sentence Tokenization with sent_tokenize(). Strengths: Good for breaking down text by sentences. Weaknesses: Less effective in irregular sentence delimiters.
  • Method 3: Punkt Sentence Tokenizer. Strengths: Handles complex sentence structures and abbreviations. Weaknesses: May require custom training for unique text styles.
  • Method 4: Regex Tokenization with RegexpTokenizer. Strengths: Highly customizable patterns. Weaknesses: Complexity increases with more intricate tokenization rules.
  • Bonus Method 5: Whitespace Tokenization with WhitespaceTokenizer. Strengths: Quick and simple to implement. Weaknesses: Does not account for punctuation or complex text structures.