Using TensorFlow to Convert Tokenized Words from the Iliad Dataset into Integers in Python

Rate this post

πŸ’‘ Problem Formulation: In natural language processing, converting textual data into a numerical format is vital for machine learning models to interpret and learn from text. Specifically, when working with the Iliad dataset, one might start with tokenized words such as [“Achilles”, “Hector”, “battle”, “Troy”] and aim to convert each unique token into a distinct integer, resulting in an output like [1, 2, 3, 4].

Method 1: Using TensorFlow’s TextVectorization Layer

This method involves using TensorFlow’s TextVectorization layer to automatically convert tokenized words into integers. This layer can be trained with the dataset’s vocabulary and will handle the mapping for you. It is especially useful for large datasets and can also handle out-of-vocabulary tokens gracefully.

Here’s an example:

import tensorflow as tf

# Sample tokenized Iliad dataset
samples = ["Achilles", "Hector", "battle", "Troy"]

# Create TextVectorization layer
vectorize_layer = tf.keras.layers.TextVectorization(output_mode='int')

# Adapt it to the dataset

# Convert tokens to integers
integer_tokens = vectorize_layer(samples)

Output: [[1], [2], [3], [4]]

This code snippet initializes a TextVectorization layer, adapts it to the provided tokenized sample data, and then uses it to convert the tokens into their corresponding integer representations. The output shows a list of integers where each element corresponds to a tokenized word from the dataset.

Method 2: Creating a Word-Index Mapping Manually

For those who want more control over the token-to-integer conversion process, creating a custom mapping dictionary may be the way to go. This manual method involves iterating over the dataset, assigning an incremental integer to each unique word.

Here’s an example:

tokens = ["Achilles", "Hector", "battle", "Troy"]
word_to_index = {word: index for index, word in enumerate(sorted(set(tokens)), start=1)}

# Convert tokens to integers
integer_tokens = [word_to_index[word] for word in tokens]

Output: [1, 2, 3, 4]

In the above code, we first create a dictionary that maps words to unique indices. Then, we convert each token into an integer using a list comprehension. The result is a list of integers that represent the original tokens.

Method 3: Using a Tokenizer Object

TensorFlow’s Tokenizer class provides a higher-level abstraction to convert words to integers. It can also keep track of the vocabulary and easily handle new, unseen words by assigning them to an ‘out of vocabulary’ token ID.

Here’s an example:

from tensorflow.keras.preprocessing.text import Tokenizer

# Initialize the tokenizer
tokenizer = Tokenizer()

# Fit the tokenizer on the texts

# Convert tokenized words to integers
sequences = tokenizer.texts_to_sequences(samples)

Output: [[1], [2], [3], [4]]

The Tokenizer object is first initialized and then fit on the sample texts which builds a word index. The texts_to_sequences method is called to convert the array of text data into an array of token integers.

Method 4: Encoding with Categorical Variables

Using TensorFlow’s tf.feature_column can be an alternative when you want to integrate the encoding into a TensorFlow data pipeline. It makes the process compatible with TensorFlow’s Dataset API, which can be beneficial for performance on large datasets.

Here’s an example:

import tensorflow as tf

# Define categorical vocabulary
vocab = ['Achilles', 'Hector', 'battle', 'Troy']
vocab_feature_column = tf.feature_column.categorical_column_with_vocabulary_list(key='tokens', vocabulary_list=vocab)

# Use indicator column to convert into integers
indicator_feature_column = tf.feature_column.indicator_column(vocab_feature_column)

# Encode sample input
input_layer = tf.feature_column.input_layer({'tokens': [['Achilles'], ['Hector'], ['battle'], ['Troy']]}, [indicator_feature_column])

Output: [[1, 0, 0, 0], [0, 1, 0, 0], [0, 0, 1, 0], [0, 0, 0, 1]]

This method creates a categorical feature column with a predefined vocabulary list and then wraps it around an indicator column that effectively encodes the strings as one-hots that can then be converted to integer indices.

Bonus One-Liner Method 5: Using Pandas Factorization

For those working with pandas DataFrames alongside TensorFlow, using pandas to factorize the tokens into integers can be a fast, one-liner solution.

Here’s an example:

import pandas as pd

token_series = pd.Series(["Achilles", "Hector", "battle", "Troy"])
integer_tokens, unique_tokens = pd.factorize(token_series)

Output: array([0, 1, 2, 3], dtype=int64)

Using pandas’ factorize function, the series of tokens is turned into an array of integer identifiers. This function also returns a second array with the unique tokens, useful for reference.


  • Method 1: TensorFlow’s TextVectorization Layer. It is integrated with TensorFlow, making it ideal for Keras models and large datasets. However, it offers less control over the tokenization process.
  • Method 2: Manual Word-Index Mapping. This gives you full control and transparency over the mapping. However, it can become cumbersome for larger vocabularies and does not handle new tokens automatically.
  • Method 3: Using a Tokenizer Object. Convenient and robust, it is part of the Keras preprocessing tools and is excellent for handling large vocabularies and out-of-vocabulary tokens. It may, however, introduce complexity when optimizing TensorFlow pipelines.
  • Method 4: Encoding with Categorical Variables. This integrates smoothly with TensorFlow data pipelines and is suitable for handling categorical data directly within TensorFlow. It is more verbose and could be less intuitive compared to other methods.
  • Bonus Method 5: Using Pandas Factorization. It is very quick and a one-liner solution when working with Pandas, but can be inefficient with memory for large datasets and requires converting between pandas and TensorFlow objects.