Does machine learning have to be hard? Well, back then when everything must be coded from scratch, yes. But today, we are stepping on the shoulders of giants. Lots of libraries are available out there to help us apply machine learning easier, so we don’t have to reinvent the wheel. In this tutorial, we are going to look at PixelLib. It is an easy-to-use library that segments objects in images and videos with five lines of code. Five lines of code for deep learning? Exactly. This article will bring you from A to Z with code snippets and video complements. So read on and see it for yourself

You can also check out this machine learning project with a video guide and advanced tips and tricks on the Finxter Computer Science Academy:

Install and Import Modules

Feel free to download the script for this tutorial from this GitHub repo. We will execute it on Google Colab and use some free GPU resources for model training. If you would like to try Google Colab out, head over to the site and sign up using your Gmail account. It looks like a Jupyter Notebook but with its storage location in your Google Drive. Upload the script onto your Google Colab and execute the blocks as we guide you in this article.

Execute the following command on a terminal or command prompt to clone any GitHub repo:

$ git clone https://github.com/username/project_name.git

The next step is to enable the GPU resource in our Colab environment. There are two ways to do this:

• Method 1: Click on the Edit tab. At the Notebook settings, choose GPU at the drop-down, and click Save.
• Method 2: Click on the Runtime tab. At the Change Runtime Type, choose GPU, and click Save.

Feel free to check out the video version of this tutorial for more in-depth explanations.

Now, install the necessary packages using pip:

$ pip install tensorflow--gpu pixellib --upgrade

As PixelLib relies on the TensorFlow library, we need to install that as well. If your execution environment has no GPU resource, omit the “–gpu” for the installation.

Now let’s import the required libraries:

import pixellib
from pixellib.instance import instance_segmentation
import matplotlib.pyplot as plt

We imported the main pixellib library as well as its instance_segmentation() function. There you go, we just executed two out of five lines of code for object segmentation. And we also imported the Matplotlib library to display images on our notebook. That’s all the libraries we need for this code project.

Although we only focus on the instance segmentation, PixelLib has much more to offer. For example, background editing, semantic segmentation, and custom training. Check out its detailed GitHub repo to learn more.

Create Instance and Load Pretrained Model

To use the instance segmentation function, we must first initialize it:

segment_image = instance_segmentation()

The initialized function is passed to a variable named segment_image. PixelLib takes a pre-trained model and performs object segmentation on a given input. It then returns a bounding box and a boundary for every detected object in the input.

We are going to use the Mask RCNN as the pre-trained model. Go ahead to this resource page at the PixelLib GitHub repo and download the .h5 model file. We will also need a sample image and a sample video to test how well PixelLib performs. If you would like to use the same resources as this tutorial, they can be downloaded from this site. Also, check out this relevant video:

Also, feel free to use your images and videos to test the library out. Once you have the model and samples ready, upload them to Google Drive.

Now, go back to your Google Colab interface and mount the Google Drive to it so that you can access the files. To do that, click on the Google Drive icon at the left of the interface for mounting – as shown in Figure 1.

You will see a folder named “drive” appear on the data repository once it is mounted. You can also upload files directly onto Google Colab, but it will not be permanent storage. Once you are off your Google Colab tab, the files will not be there the next time you open them.

Now you have access to the files. Time to reveal the fourth line of code:

segment_image.load_model("/content/drive/MyDrive/mask_rcnn_coco.h5")

The load_model() function takes a string path and uses the model for the segmentation work. Your folder location might be different than the sample code, so adjust accordingly.

Object Segmentation on Images

Let’s display the sample image using matplotlib and see how it looks like:

plt.figure(figsize=(26, 18))
im_bike_lane = plt.imread("/content/drive/MyDrive/bike-lane.jpg")
plt.imshow(im_bike_lane)

We defined a plotting figure of size 26 inches in width and 18 inches in height. Then, we loaded and displayed the sample image using the imread() and imshow() functions. Figure 2 shows the image, which is a normal street view consisting of people, cars, bicycles, etc.

OK, now comes the code for instance segmentation. Execute the following line:

segment_image.segmentImage("/content/drive/MyDrive/bike-lane.jpg",
       show_bboxes=True, 
       extract_segmented_objects=False, 
       save_extracted_objects=False, 
       output_image_name="/content/drive/MyDrive/ bike-lane_segmented.jpg")

We set the configurations for segmentImage() and let it do the heavy-lifting. The show_bboxes config is set to True because we want every detected object to having a bounding box. We set extract_segmented_objects to False to not extract objects as individual images. The save_extracted_objects is set to False to not save objects as separate files. The output_image_name config is the path where you want the new image to be saved. Figure 3 shows the processed image with all detected objects.

Not bad! Without model training and tuning, PixelLib managed to detect objects from the image. Categories like cars, bicycles, persons and traffic lights are detected. If you chose to save object segments, you will see object crop-outs saved as individual files.

What if we want to segment only one, or a handful of categories instead of all categories? That can also be easily configured, as follows:

target_classes = segment_image.select_target_classes(person=True)
#or
target_classes = segment_image.select_target_classes(car=True,bicycle=True)

segment_image.segmentImage("/content/drive/MyDrive/bike-lane.jpg", 
      segment_target_classes=target_classes,
      show_bboxes=True, 
      extract_segmented_objects=False, 
      save_extracted_objects=False, 
      output_image_name="/content/drive/MyDrive/bike-lane_segmented.jpg")

We gave the select_target_classes() function the category names to segment for. The configuration is passed to the target_classes variable. The first target_classes variable has config “person=True”. Only objects that are predicted as Person in the image will be segmented. Likewise, the second target_classes variable sets Car and Bicycle categories as True. That means we want to segment only these two objects. Figure 4 shows the segmented image with the target class Person. Figure 5 shows the one with target classes Car and Bicycle.

So, what do you think about the object segmentation ability of PixelLib? Impressive, isn’t it? Objects are detected accurately according to the labels. Boundaries are defined precisely around the detected objects. Some objects are detected accurately even when half of their shapes are occluded.

Object Segmentation on Videos

So far, so good on images. Now let’s see how PixelLib performs for video segmentation. Figure 6 shows a screenshot of the video, which is another street view with humans and vehicles.

The code for video instance segmentation is like image segmentation. Surely you get the drill now with the straightforward function names. Execute the following code:

segment_video = instance_segmentation()
segment_video.load_model("/content/drive/MyDrive/mask_rcnn_coco.h5")
segment_video.process_video("/content/drive/MyDrive/city-scene.mp4", 
       show_bboxes=True,  
       extract_segmented_objects=True,
       save_extracted_objects=False, 
       frames_per_second= 5,  
       output_video_name="/content/drive/MyDrive/city-scene_segmented.mp4")

We initialized the instance_segmentation() function again and passed it to the segment_video variable. The same pre-trained model is loaded using the load_model() function. We used the process_video() function and gave it the same configs. The additional frames_per_second config is the frame rate of the video to be saved.

You might realize that it took a long time for PixelLib to process the video compared to the image. That is normal because videos are a collection of images. PixelLib processes the video frame by frame and performs object segmentation. Figure 7 shows a screenshot of how a segmented video frame looks like.

As expected, the library accurately detected objects in the video. That’s a ready-to-use object segmentation library without any hyperparameter tweaking! If you think PixelLib is useful, check out its repo to explore more functionalities.

Conclusion

That’s it! You have learned that there exists a well-performing object segmentation library. And it can be implemented in five lines of code. Another useful tool to add to your machine learning skillset. I hope this was a fun learning process for you! If you encounter any issues and would like an in-depth walkthrough of the code, the video explanation is there to help you out. Happy learning!

You can check out this tutorial with a video guide and advanced guidance on the Finxter Computer Science Academy:

Are you looking for interesting deep learning projects that are suitable for beginners? Do not worry, this is not another MNIST image classification tutorial. Instead, we are going to classify some Star Wars Lego images using the TensorFlow library. This tutorial will sharpen your knowledge about convolutional neural networks and transfer learning. Intrigued? Let’s get started.

Install and Import Modules

Feel free to download the script for this tutorial from this GitHub repo. We will execute it in Google Colab and use some free GPU resources for model training. If you would like to try Google Colab out, head over to the site and sign up using your Gmail account. It looks like Jupyter Notebook but with its storage location in your Google Drive. Upload the script onto your Google Colab and execute it along as you read through this article.

Execute the following command on a terminal or command prompt to clone any GitHub repo:

$ git clone https://github.com/username/project_name.git

The second step is to enable the GPU resource in our Colab environment. There are two ways to do this:

• Method 1: Click on the Edit tab. At the Notebook settings, choose GPU at the drop-down, and click Save.
• Method 2: Click on the Runtime tab. At the Change Runtime Type, choose GPU, and click Save.

Feel free to check out the video version of this tutorial for more in-depth explanations.

Now, install the necessary packages using pip:

$ pip install numpy pandas matplotlib seaborn tensorflow

As well as importing all the required libraries:

import os
import math
import random
import shutil

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sn

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

We need the first four libraries to restructure data folders. The Numpy and Pandas libraries will be used for data manipulation. The Matplotlib and Seaborn libraries will be used to display plots and figures. The TensorFlow library is what we will use for machine learning modeling.

Note: We heavily use the keras library in this tutorial. Keras simplifies the implementation of deep learning models with TensorFlow.

If you’re interested in Keras, feel free to watch this video and read the associated blog article on the income levels of Keras developers:

To check if GPU is enabled in our notebook environment, execute the following:

tf.test.gpu_device_name()

If a GPU is allocated for your notebook, you will see a printout like ‘/device: GPU:0’ instead of an empty string.

Execute the following line to check which GPU is assigned for you:

!nvidia-smi

Locate the GPU name at the printout. For example, ‘Tesla P4’.

Load Dataset and Preprocess Data

So far, so good. Now, we need to download the dataset, which is the Lego Minifigures dataset from Kaggle. We are going to use only the Star Wars folder for this tutorial. There are 15 subfolders of images in the Star Wars folder. To simplify things, we are going to use only the first five folders out of 15. So, go ahead and remove everything except for the first five folders of Star Wars. Then, upload this data folder to your Google Drive (the one that has the same Gmail account as your Colab).

Dataset uploaded. Now, go back to our Google Colab interface and mount the Google Drive to it so that we can access the data. To do that, click on the Google Drive icon at the left of the interface to mount it – as shown in Figure 1.

You will see a folder named “drive” appear on the data repository once it is mounted. Next, we are going to restructure our data folder. We want to create a train set, a validation set, and a test set for modeling and evaluation.

Execute the following code to restructure the data folder:

BASE_DIR = '/content/drive/MyDrive/star-wars/'
names = ["YODA", "LUKE SKYWALKER", "R2-D2", "MACE WINDU", "GENERAL GRIEVOUS"]
train_proportion = 0.6
val_proportion = 0.25
total_train = 0
total_val = 0
total_test = 0

In the code, we defined a base directory for the Google Drive folder. We also reassigned names to the five data subfolders. We defined the proportion of image distribution in each folder to be copied onto the new sets. For example, in a subfolder of 10 images, 6 images will be copied to the train set, 3 to the validation set, and the remaining images to the test set. The variables total_train, total_val, and total_test are counters to calculate the total number of images in the said folders.

Execute the following lines to create new folders:

if not os.path.isdir(f'{BASE_DIR}train/'):
    for name in names:
        os.makedirs(f'{BASE_DIR}train/{name}')
        os.makedirs(f'{BASE_DIR}val/{name}')
        os.makedirs(f'{BASE_DIR}test/{name}')

Executing the following code will copy the images over to the new folders:

orig_folders = ["0001/", "0002/", "0003/", "0004/", "0005/"]

for folder_idx, folder in enumerate(orig_folders):
    files = os.listdir(BASE_DIR + folder)

    folder_name = names[folder_idx]
    number_of_images = len([name for name in files])

    n_train = int((number_of_images * train_proportion) + 0.5)
    n_valid = int((number_of_images * val_proportion) + 0.5)
    n_test = number_of_images - n_train - n_valid

    total_train += n_train
    total_val += n_valid
    total_test += n_test

    print(f'Folder {folder_name} has {number_of_images} images in total:\n train - {n_train}, val - {n_valid}, test - {n_test}\n')

    # copy images from original folders to the new ones
    for idx, file in enumerate(files):
        file_name = BASE_DIR + folder + file
        if idx < n_train:
            shutil.copy(file_name, f'{BASE_DIR}train/{folder_name}')
        elif idx < n_train + n_valid:
            shutil.copy(file_name, f'{BASE_DIR}val/{folder_name}')
        else:
            shutil.copy(file_name, f'{BASE_DIR}test/{folder_name}')

The for loop iterates through each subfolder and copy the images to the new folders. You will see that three new data folders are created, each containing five image folders.

We can now proceed to the data pre-processing step. One of the common pre-processing techniques for image data is normalization. It is to squeeze the value of input images from a range of 0-255 to a range of 0-1. That usually helps for better model training and convergence.

Execute the following lines to create three data generators:

train_gen = keras.preprocessing.image.ImageDataGenerator(rescale=1./255, 
	     rotation_range=20, horizontal_flip=True, 
             width_shift_range=0.2, height_shift_range=0.2, 
             shear_range=0.2, zoom_range=0.2)
valid_gen = keras.preprocessing.image.ImageDataGenerator(rescale=1./255)
test_gen = keras.preprocessing.image.ImageDataGenerator(rescale=1./255)

Data generators load data in batch, perform data pre-processing and pass them to a machine learning model. The rescale=1./255 configuration is for image normalization. More pre-processing techniques are also configured for the train generator.

Next, execute the following lines to load the data in batches using data generators:

target_size = (256, 256)
batch_size = 4

train_batches = train_gen.flow_from_directory(
    f'{BASE_DIR}train',
    target_size=target_size,
    class_mode='sparse',
    batch_size=batch_size,
    shuffle=True,
    color_mode="rgb",
    classes=names   
)

val_batches = valid_gen.flow_from_directory(
    f'{BASE_DIR}val',
    target_size=target_size,
    class_mode='sparse',
    batch_size=batch_size,
    shuffle=False,
    color_mode="rgb",
    classes=names
)

test_batches = test_gen.flow_from_directory(
    f'{BASE_DIR}test',
    target_size=target_size,
    class_mode='sparse',
    batch_size=batch_size,
    shuffle=False,
    color_mode="rgb",
    classes=names
)

We define the target input size as (256, 256) with a batch size of 4. The flow_from_directory() function of the data generators pulls the data from the given directory one batch at a time.

The rest of the code in this section is to check how data batches look like. Execute the code in the script and see if the outcome makes sense.

Train and Evaluate A CNN Model

It seems like we spent a lot of time fiddling with the data. That is where most data scientists spent their time. Good data processing is crucial for training machine learning models. You will thank yourself for the effort later.

Now it’s time to get our hands dirty for machine learning! Execute the following lines to create a basic convolutional neural network model:

model = keras.models.Sequential()
model.add(layers.Conv2D(32, (3,3), strides=(1,1), padding="valid", 
activation='relu', input_shape=(256, 256,3)))
model.add(layers.MaxPool2D((2,2)))
model.add(layers.Conv2D(64, 3, activation='relu'))
model.add(layers.MaxPool2D((2,2)))
model.add(layers.Flatten())
model.add(layers.Dense(64, activation='relu'))
model.add(layers.Dense(5))

If you want to see how the model architecture looks like, execute this line:

model.summary()

From the printout, we see that the model consists of two convolution layers and max-pooling layers, as well as a flatten layer. It is followed by a dense layer and an output layer of 5 units. The unit of output layer determines the number of categories for model prediction.

Let’s compile the model with the loss, accuracy, and optimization functions:

loss = keras.losses.SparseCategoricalCrossentropy(from_logits=True)
optim = keras.optimizers.Adam(learning_rate=0.001)
metrics = ["accuracy"]

model.compile(optimizer=optim, loss=loss, metrics=metrics)

Note that there is no one-hot encoding for the labels (and no Softmax at the output layer). The loss function is chosen based on that.

Execute the following lines for the actual model training:

epochs = 30

early_stopping = keras.callbacks.EarlyStopping(monitor="val_loss", 
patience=5, verbose=2)

history = model.fit(train_batches, validation_data=val_batches,
callbacks=[early_stopping],epochs=epochs, verbose=2)

model.save(f"{BASE_DIR}lego_model.h5")

We defined 30 epochs and an EarlyStopping() function to the fit() function. The EarlyStopping() function will end the model training when the criteria are fulfilled. The fit() function trains and validates the model. We also saved the model as an H5 file when the training is completed.

Let’s check out how the model performs by plotting the loss and accuracy outcomes:

plt.figure(figsize=(16, 6))
plt.subplot(1, 2, 1)
plt.plot(history.history['loss'], label='train loss')
plt.plot(history.history['val_loss'], label='valid loss')
plt.grid()
plt.legend(fontsize=15)

plt.subplot(1, 2, 2)
plt.plot(history.history['accuracy'], label='train acc')
plt.plot(history.history['val_accuracy'], label='valid acc')
plt.grid()
plt.legend(fontsize=15)

As shown in Figure 2, most of the train and validation losses are in the high range of 1.0 and 2.0. The train and validation accuracies fluctuated a lot. Generally, the model outcome is not optimal.

We can see the model performance more obvious with images and predictions. Execute the following lines to make predictions with test data and plot the outcome:

predictions = model.predict(test_batches)
predictions = tf.nn.softmax(predictions)
labels = np.argmax(predictions, axis=1)

print(test_batches[0][1])
print(labels[0:4])

show(test_batches[0], labels[0:4])

Note that Softmax is added here to compare the result with respective labels. This is how the prediction looks like:

As we can see, the model predicted all four sample test data incorrectly. It categorized most of the inputs as R2-D2, which shows a sign of overfitting. So, how do we go about improving the model performance?

Comparison with a Transfer Learning Model

There are many things that we can do to improve the model. An example would be to adjust the hyperparameters. We will instead replace the model architecture with a transfer learning model. Let’s see if this can yield a better model performance.

Create a transfer learning model based on the VGG16 architecture:

vgg_model = tf.keras.applications.vgg16.VGG16()

model = keras.models.Sequential()
for layer in vgg_model.layers[0:-1]:
    model.add(layer)

for layer in model.layers:
    layer.trainable = False

Here, we download the pre-trained weights of the VGG16 model. We add all layers except the output layer to a new model and make the weights unchangeable. Then, we add an output layer to the model, as follows:

model.add(layers.Dense(5))

That’s it, now we have a transfer learning model with a custom output layer. We only need to train the last layer while all other layers stay the same. Compile the model with the same configurations as the basic CNN model:

loss = keras.losses.SparseCategoricalCrossentropy(from_logits=True)
optim = keras.optimizers.Adam(learning_rate=0.001)
metrics = ["accuracy"]

model.compile(optimizer=optim, loss=loss, metrics=metrics)

The VGG16 function from TensorFlow comes with its pre-processing function. We use it for all the data generators:

preprocess_input = tf.keras.applications.vgg16.preprocess_input

train_gen = keras.preprocessing.image.ImageDataGenerator(preprocessing_function=preprocess_input)
valid_gen = keras.preprocessing.image.ImageDataGenerator(preprocessing_function=preprocess_input)
test_gen = keras.preprocessing.image.ImageDataGenerator(preprocessing_function=preprocess_input)

Train the model using the same workflow we defined earlier, as follows:

target_size = (224, 224)
batch_size = 4

train_batches = train_gen.flow_from_directory(
    f'{BASE_DIR}train',
    target_size=target_size,
    class_mode='sparse',
    batch_size=batch_size,
    shuffle=True,
    color_mode="rgb",
    classes=names   
)

val_batches = valid_gen.flow_from_directory(
    f'{BASE_DIR}val',
    target_size=target_size,
    class_mode='sparse',
    batch_size=batch_size,
    shuffle=False,
    color_mode="rgb",
    classes=names
)

test_batches = test_gen.flow_from_directory(
    f'{BASE_DIR}test',
    target_size=target_size,
    class_mode='sparse',
    batch_size=batch_size,
    shuffle=False,
    color_mode="rgb",
    classes=names
)

epochs = 30

# callbacks
early_stopping = keras.callbacks.EarlyStopping(
    monitor="val_loss",
    patience=5,
    verbose=2)

history = model.fit(train_batches, 
                    validation_data=val_batches,
                    callbacks=[early_stopping],
                    epochs=epochs, verbose=2)

model.save(f"{BASE_DIR}lego_model_transfer-learning.h5")

Done with model training. Now we plot its losses and accuracies, as well as testing the model with the same batch of test data:

plt.figure(figsize=(16, 6))
plt.subplot(1, 2, 1)
plt.plot(history.history['loss'], label='train loss')
plt.plot(history.history['val_loss'], label='valid loss')
plt.grid()
plt.legend(fontsize=15)

plt.subplot(1, 2, 2)
plt.plot(history.history['accuracy'], label='train acc')
plt.plot(history.history['val_accuracy'], label='valid acc')
plt.grid()
plt.legend(fontsize=15);

model.evaluate(test_batches, verbose=2)

# make some predictions
predictions = model.predict(test_batches)
predictions = tf.nn.softmax(predictions)
labels = np.argmax(predictions, axis=1)

print(test_batches[0][1])
print(labels[0:4])

show(test_batches[0], labels[0:4])

As shown in Figure 5, the model got three out of four sample data right. The training loss in Figure 4 looks better than the previous model. Both the train and validation accuracies achieved a higher and less fluctuating outcome. So we can say that the transfer learning model performs better than the basic CNN model. Note that both the models can be further optimized, so do not take this code example as an end-all result.

Conclusion

Yay! We learned about image classification by implementing a basic CNN and a transfer learning CNN. I hope this was a fun learning process for you! If you encounter any issues and would like an in-depth walkthrough of the code, the video explanation is there to help you out. Happy learning!

The OpenCV [1] library contains most of the functions we need for working with images. Handling images in programming requires a different intuition than handling text data. An image is made up of pixels. It looks like a spreadsheet full of cells with numerical values when zoomed in. Each pixel usually contains a value ranging between 0 to 255. The value indicates the degree of brightness for the color it is assigned to. So, how do we work with images in Python? We first need to load them as NumPy arrays, converting all image pixels into numerical values. Only then we can use different computer vision techniques to manipulate them.

In this article, we are going to get our hands dirty experimenting with images using OpenCV. We will look at techniques like color conversion, resizing, blending, blurring, and thresholding. Getting your image data right is a half-way success for a useful machine learning model. Intrigued? Let’s get started.

Install and Import Required Modules

For this tutorial, we need to install the OpenCV, NumPy, and Matplotlib modules. NumPy is used to manipulate image arrays. Matplotlib is used to display images for comparing the “before and after”. Feel free to clone the GitHub repo of this tutorial.

First, create a virtual environment for this project. Then, install the mentioned modules in a Jupyter notebook:

!pip install opencv-python  
!pip install numpy
!pip install matplotlib

No surprise here — the installation should be straightforward and fast. Now execute the following lines of code in your notebook:

import cv2
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

Note that the %matplotlib inline magic command is exclusive for Jupyter notebooks. It is not required in a Python script. It sets the backend of the Matplotlib module to display figures inline and not on a separate window.

Done! Get your favorite photos ready — it’s time for experiments!

Load Image and Convert Color Channels

To load an image in the notebook, we use the imread method of the OpenCV module. By default, the method loads an image in color. To load a greyscale image, we need to supply a second parameter of “0’” to the method:

img_greyscale = cv2.imread('./photo.jpg', 0)
img_greyscale.shape
img = cv2.imread('./photo.jpg')
img.shape

Note that the images are loaded as NumPy arrays – one greyscale and another one in color. The shape method returns (5563, 3709) for the variable img_greyscale and (5563, 3709, 3) for img. The method returns information in the form of (height, width, channel). Both the variables have the same height and width values. But img_greyscale consists of only one channel (one color) while img has three.

By default, the imread method loads an image with a color order of blue, green, red. It is not the usual red, green, blue. In case you ever wonder why your images look weird in OpenCV, it is that. To display an image, use the imshow method of the Matplotlib module as follows:

plt.imshow(img)

Figure 1 shows how different an image can look when its color channels are mixed up. Matplotlib displays the red channel as blue for the image on the left. To fix this, we can use the OpenCV cvtColor method to convert the color channels from (B, G, R) to (R, G, B), as follows:

img_RGB = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)

The color-corrected image is shown on the right side of Figure 1. We will use the RGB image as an example in the latter sections. But using RGB images is not a requisite — feel free to use the BGR image if you prefer. Make sure to pick the correct channels in operation.

Resize Image

Quiz time: which OpenCV method should you use to resize an image? You guessed it — the resize method. It takes an image and an image dimension as parameters. The following code resizes the image to be half its original size:

width = int(img_RGB.shape[1] / 2)
height = int(img_RGB.shape[0] / 2)

img_RGB_smaller = cv2.resize(src=img_RGB, dsize=(width, height)) 
img_RGB_smaller.shape

Note that you can supply any positive integer values to the dsize parameter of the resize method. Yet, it is a good practice to use a scale factor to keep the original aspect ratio of the image. The code shown takes the width and height values of the original image and divides them by two. The output of the img_RGB_smaller.shape is (2781, 1854, 3), which is 50% smaller than its original size, (5563, 3709, 3). You can also make the image larger by multiplying its width and height by two, as follows:

width = int(img_RGB.shape[1] * 2)
height = int(img_RGB.shape[0] * 2)

img_RGB_bigger = cv2.resize(src=img_RGB, dsize=(width, height)) 
img_RGB_bigger.shape

That creates an image of size (11126, 7418, 3). Feel free to be creative with the image dimension definitions. Figure 2 shows the resized images. Both look the same because their aspect ratios are retained. Note the differences in their width and height axes instead.

Blend Images

Image blending means combining two images with shared transparency. We want two images to “blend into” each other as one image. For this, we need to load another image to our notebook:

img_overlay = cv2.imread('./photo-overlay.jpg')
img_overlay.shape

All images used in this code project can be found at Unsplash.com. The second image is loaded as variable img_overlay with dimensions (2000, 1800, 3). Images must have the same size for image blending. As img_overlay is of a different size than the first image, we need to resize it to match the size of the first image:

img_overlay = cv2.resize(img_overlay, (img_RGB.shape[1], img_RGB.shape[0]))
img_overlay.shape

Note that the dsize parameter takes a value in the form of (width, height), not (height, width). Thus, we enter (img_RGB.shape[1], img_RGB.shape[0]) as the parameter instead of the other way round. Now, the output of img_overlay.shape should show the same size as img_RGB, which is (5563, 3709, 3). Enter the following code to blend both the images together:

blended = cv2.addWeighted(src1=img_RGB, alpha=0.3, src2=img_overlay, beta=0.7, gamma=0)

The addWeighted method of OpenCV combines the images with a “transparency weightage”. The src1 parameter takes the background image and the src2 the foreground image. The alpha parameter sets the transparency of src1 and the beta of src2. Both alpha and beta can take values ranging from 0 to 1 and should both add up to 1. A value closer to 0 indicates more transparency. A value closer to 1 indicates more opaqueness. The gamma parameter sets the brightness of the output image. Figure 3 shows the before and after of the image blending operation.

Blur Image

Here, we crop out a smaller section of an image to better notice the image blurring operation. Taking img_RGB, we copy its bottom right part as img_small using NumPy array slicing:

img_small = img_RGB[4000:, 2000:]

That will create a smaller image of size (1563, 1709, 3). There are various image blurring functions in the OpenCV module. For example, average blurring, median blurring, and Gaussian blurring. They differ in their mathematical operations and outcomes. For the sake of simplicity, we use the basic average blurring function in this tutorial. Enter the following line of code in your notebook:

blurred = cv2.blur(src=img_small, ksize=(100, 100))

You must have familiarized yourselves with the OpenCV parameters by now. Otherwise, press the SHIFT + TAB buttons to view any function description. The ksize parameter of the blur method defines the dimensions of the filter kernel. A kernel is like a paintbrush or sponge that you use to “smudge” the original image and make it blurry. The ksize parameter is the width and height of the sponge that you want to use – in this case, 100 x 100. Figure 4 shows the cropped image with its blurred after effect.

Threshold Image

Image thresholding turns a greyscale image into either black or white pixels. You might be asking: what’s the need for blurring and thresholding images? The answer is: so that computational models can perceive image data better. Take edge detection as an example: we want to blur or smooth object edges so that there will be less noise. And we want to threshold images so object boundaries can be defined better.

For thresholding, we use img_greyscale instead of the coloured image. Enter the following one-liner in your notebook:

ret, thresh1 = cv2.threshold(src=img_greyscale, thresh=127, maxval=255, type=cv2.THRESH_BINARY)

The threshold method takes a greyscale image as its src parameter. The thresh parameter is the cutting-point for the black/white pixel decision. Any pixel value lower than the thresh value will be assigned 0. Any pixel value above the thresh value will be assigned 1. That creates the black-or-white contrast. As the image has its values ranged from 0 to 255, we assign the maxval (largest value) parameter as 255. The type parameter defines the kind of threshold we want. THRESH_BINARY converts all shades of grey in the image into either black or white. Figure 5 shows a greyscale image with its outcome after the thresholding operation.

You have just learned five useful techniques in computer vision. Well done!

Conclusion

This article elaborates on five basic image processing techniques of OpenCV. They include color conversion, resizing, blending, blurring, and thresholding. It is a step-by-step introductory tutorial to perform computer vision operations in Python.

References

[1] https://opencv.org/about/

First released in 2007, PyTesseract [1] is the to-go library for extracting text from images. It uses classical computer vision methods to perform optical character recognition (OCR), then features neural network components such as LSTM from its fourth version onwards.

You may ask: Is there any alternative that is as good as PyTesseract for OCR? Yes, EasyOCR [2] it is. It is a new, deep learning-based module for reading text from all kinds of images in more than 80 languages.

In this article, we will go through a three-step tutorial.

• First, we will install the required libraries.
• Second, we will perform image-to-text processing using EasyOCR on various images.
• Third, we will use OpenCV to overlay detected texts on the original images. Let’s get started.

Step 1: Install and Import Required Modules

Optical character recognition is a process of reading text from images. An easy task for humans, but more work for computers to identify text from image pixels. For this tutorial, we will need OpenCV, Matplotlib, Numpy, PyTorch, and EasyOCR modules. Here’s the GitHub repo of this tutorial. 

You can follow the tutorial in our interactive Jupyter notebook online:

First, create a virtual environment for this project. Then, install the mentioned modules in a Jupyter notebook:

!pip install opencv-python
!pip install matplotlib
!pip install numpy
!pip install torch==1.7.1+cpu torchvision==0.8.2+cpu torchaudio===0.7.2 -f https://download.pytorch.org/whl/torch_stable.html
!pip install easyocr

The OpenCV module is for computer vision-related operations in Python. Specifically, we will use it to overlay images with respective recognized texts later. We need the Matplotlib module to display images. And we will use the Numpy module to convert images into arrays.

PyTorch is a prerequisite for the EasyOCR module. Its installation varies according to OS and GPU driver requirements. You can get the installation commands at the PyTorch homepage [3]. Copy and execute the respective command as shown in Figure 1 if you operate on Windows.

Now, go ahead and install the EasyOCR module — the tool we need for extracting text from images. At this point, you should be able to execute the following lines of code in your notebook:

import cv2
import numpy as np
import easyocr
import matplotlib.pyplot as plt
%matplotlib inline

Note that the %matplotlib inline magic command is exclusive for Jupyter notebooks. It is not required in a Python script. It sets the backend of the Matplotlib module to display figures inline and not on a separate window.

You’re off to a great start! Now, onto the next step.

Step 2: Load Images and Extract Text using EasyOCR

For copyright reasons, all images used in the sample notebook are not provided in the GitHub repo. Feel free to download them from Unsplash.com or use your images. Define the path of an image using the following code:

im_1_path = './folder/image_name.jpg'

Next, initialize an EasyOCR reader with a list of languages you would like to use. Use the reader to read an image with the following function:

def recognize_text(img_path):
    '''loads an image and recognizes text.'''
    
    reader = easyocr.Reader(['en'])
    return reader.readtext(img_path)

Did it surprise you that two lines of code are all you need to perform OCR? “Easy” for EasyOCR! The recognize_text() function initializes an OCR reader to a variable named reader. It takes a list of languages as a parameter. For this tutorial, we want to only recognize English text, thus the ‘en’ in the list. The readtext method reads an image given its stored directory. The returned OCR result is passed as the output of the recognize_text() function.

result = recognize_text(im_1_path)
result

Note that it is going to take more time to execute EasyOCR on a CPU instead of a GPU. The im_1_path image took around ten seconds to be executed by recognize_text(). Figure 2 shows the operations in the EasyOCR framework. The framework includes image preprocessing, deep learning model recognition, and image postprocessing.

Here is the output of the EasyOCR module:

[([[1421, 1139], [1453, 1139], [1453, 1177], [1421, 1177]],
  'S',
  0.8625819477165351),
 ([[1524, 1038], [2201, 1038], [2201, 1211], [1524, 1211]],
  'CCC444',
  0.9068348515895301),
 ([[1641, 1201], [2012, 1201], [2012, 1245], [1641, 1245]],
  'T E S L A.C O M',
  0.33458756243407134),
 ([[2519, 1254], [2790, 1254], [2790, 1284], [2519, 1284]],
  'DUAL MSTOF',
  0.24584700695087508)]

It returns a list of detected text, with each text element containing three types of information. Which are: the text, its bounding box vertices, and the confidence level of the text detection. From the output, EasyOCR detected four text elements: ‘S’, ‘CCC444’, ‘T E S L A.C O M’, and ‘DUAL MSTOF’.

To check the accuracy of the OCR, we need to display the original image on our notebook:

img_1 = cv2.imread(im_1_path)
img_1 = cv2.cvtColor(img_1, cv2.COLOR_BGR2RGB)
plt.imshow(img_1)

The imread method of the OpenCV module loads an image as a Numpy array, which is assigned to the img_1 variable. The default color channels of OpenCV are (Blue, Green, Red) instead of (Red, Green, Blue). That is why we use the cvtColor method for channel conversion. Otherwise, we will see the image with its blue color assumed as red and vice versa. The image is shown in Figure 3, which is a car with a rear view of its vehicle registration plate.

Comparing the image with its OCR output, the car plate is captured accurately. EasyOCR detects the country code and the car provider name. Yet, the ‘DUAL MOTOR text on the right side of the car is detected as ‘DUAL MSTOF’. For that, image pre-processing techniques can be used to increase the OCR accuracy. But for now, we will only test the performance of EasyOCR out of the box.

Step 3: Overlay Recognized Text on Images using OpenCV

Now, we want to draw a rectangle around each recognized text element on its original image. The overlay_ocr_text() function will be explained task-by-task.

def overlay_ocr_text(img_path, save_name):
    '''loads an image, recognizes text, and overlays the text on the image.'''
    
    # loads image
    img = cv2.imread(img_path)
    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
    
    dpi = 80
    fig_width, fig_height = int(img.shape[0]/dpi), int(img.shape[1]/dpi)
    plt.figure()
    f, axarr = plt.subplots(1,2, figsize=(fig_width, fig_height)) 
    axarr[0].imshow(img)

First, we use the OpenCV module to load an image as a Numpy array and correct its color channels. The array is assigned to the variable img. We want to display two images — the original image, and the original image with recognized texts. The subplots method of Matplotlib is used to display more than one figure at a time. The imshow method of the axarr[0] variable displays the original image.

# recognize text
    result = recognize_text(img_path)

    # if OCR prob is over 0.5, overlay bounding box and text
    for (bbox, text, prob) in result:
        if prob >= 0.5:
            # display 
            print(f'Detected text: {text} (Probability: {prob:.2f})')

            # get top-left and bottom-right bbox vertices
            (top_left, top_right, bottom_right, bottom_left) = bbox
            top_left = (int(top_left[0]), int(top_left[1]))
            bottom_right = (int(bottom_right[0]), int(bottom_right[1]))

            # create a rectangle for bbox display
            cv2.rectangle(img=img, pt1=top_left, pt2=bottom_right, color=(255, 0, 0), thickness=10)

            # put recognized text
            cv2.putText(img=img, text=text, org=(top_left[0], top_left[1] - 10), fontFace=cv2.FONT_HERSHEY_SIMPLEX, fontScale=1, color=(255, 0, 0), thickness=8)

The recognize_text() function returns the OCR output and assigns it to the result variable. A for loop is created to go through each text element contained in the variable. Recognized text elements are displayed only if their OCR confidence levels are higher than 0.5 (prob >= 0.5). Then, the top left and bottom right vertices of each bounding box are obtained. They are converted into tuples of integer values (as required by OpenCV).

The rectangle method creates a green bounding box for each detected text element. The putText method displays recognized text above its respective bounding box. As all these are done in a for loop, the operation repeats for every recognized text in the result variable.

# show and save image
    axarr[1].imshow(img)
    plt.savefig(f'./output/{save_name}_overlay.jpg', bbox_inches='tight')

Finally, the overlay_ocr_text() function displays every created text and bounding box. The imshow method of the axarr[1] variable displays the final image. As both the left and right images are in the same subplot, they are displayed as one final image. The savefig method stores the final image to a defined local directory.

How Well Did EasyOCR Perform?

The figures below show how well EasyOCR performs for different kinds of images. We will test the library on handwriting, digits, an electronic invoice, and a public sign. For a complete overview, please refer to the demo notebook in the given GitHub repo.

EasyOCR detects most of the text in Figure 7 correctly, except the text on the right-hand side.

EasyOCR manages to detect every text in Figure 5. But the text sequence is not entirely correct.

EasyOCR detects everything in Figure 6 correctly. It is a relatively large image with clear printed digits and texts, which makes the OCR perform better.

EasyOCR manages to detect every text on the invoice accurately without image preprocessing.

Again, EasyOCR nails it for Figure 8. Every text on the figure is correctly detected.

We had the impression that EasyOCR performs greatly on images with clear text. It works fine without having to preprocess images, which saves time and cost.

Bonus: Text-to-Speech Recognition

Outputs from OCR can be further utilized with a simple text-to-speech recognition application. It converts text to a voice utterance. First, we need to install the PyTTSX3 [4] module as follows:

!pip install pyttsx3

The implementation can be done in five lines of code:

import pyttsx3

engine = pyttsx3.init()
engine.setProperty('rate', 100)
engine.say(sentence)
engine.runAndWait()

The code initializes a TTS engine and assigns it to the variable engine. The setProperty method defines the speed of the utterance. The say method registers the text sentence to be pronounced. Finally, the runAndWait method executes the text-to-speech operation.

Conclusion

This article explains how to extract text elements from images using EasyOCR. It also shows how to overlay recognized text on images using OpenCV. A simple text-to-speech is also introduced as an extended application for OCR output.

References

[1] https://github.com/madmaze/pytesseract

[2] https://github.com/JaidedAI/EasyOCR

[3] https://pytorch.org/get-started/locally/

[4] https://pypi.org/project/pyttsx3/

Programming Humor

There are various Python tools available to create web applications and frontend GUIs. For example, Flask and Django. As useful as they are, we still need to invest time in learning new frameworks. That might not be the highest on our priority list when we have spent a lot of time doing our Python project.

Is there a way to create a simple, interactive web application with basic Python skills? The answer is yes, and this article will show you exactly that in five steps.

Your PDF Certificate: If you want to get a PDF certificate for completing this small course — e.g., to show to your employer or freelancing clients — feel free to check this out as a course on the Finxter Academy!

We will first create some interactive widgets using ipywidgets in a Jupyter notebook. Then, we will render the notebook as a web application using Voilà. We will push the project onto a GitHub repository, and finally, host the web app on Binder.

Let’s get started!

Step One: Create A New Git Repository and Install Python Packages

It is unnecessary to push every coding project onto our Git repository. But we need a Git repo for this project. That is because Binder will refer to it to create a Docker container that hosts our web application. For this tutorial, I have created a GitHub repo named jupyter_webapp_demo. Feel free to clone it at https://github.com/finxter/jupyter_webapp_demo.

To clone a Git repo onto your local directory, type the following command on a terminal or command prompt:

$ git clone https://github.com/username/project_name.git

Next, we are going to create a virtual environment for the code project. It is always a good practice to isolate the required dependencies per project. We will export the names and versions of all installed packages to a text file later as a requisite for Binder.

To create and activate a virtual environment, type the following commands:

$ python3 -m venv venv_name
$ cd venv_name\Scripts
$ activate
$ cd ..\..

Figure 1: Clone a Git repository and activate a virtual environment.

Install the Jupyter Notebook app using either the conda or the pip package manager:

$ conda install -c conda-forge notebook

or

$ pip install notebook

Type the following command to open the Jupyter Notebook app:

$ jupyter notebook

You will see it opened on your default web browser.

Now, create a new Python 3 notebook. Install three modules named widgetsnbextension, ipywidgets, and voila, as follows:

$ pip install widgetsnbextension 
$ pip install ipywidgets 
$ pip install voila

We need to enable the extensions to be properly displayed on the notebook:

!jupyter nbextension enable --py widgetsnbextension --sys-prefix
!jupyter serverextension enable voila --sys-prefix

Comment out the commands and execute the cell again. That will hide the displayed text from the previous cell execution from our web app.

#!jupyter nbextension enable --py widgetsnbextension --sys-prefix
#!jupyter serverextension enable voila --sys-prefix

Now we are good to go. Time to load the web app with some sliders and buttons.

Step Two: Create Interactive Web Elements with Ipywidgets

Ipywidgets is a module that lets us create interactive widgets in Jupyter notebooks. For example, buttons, text boxes, sliders, progress bars, and more. Feel free to explore the widget list [1] and use any that would help in showcasing your code project.

Now, let’s create some widgets that curate a text message to Grandma for a dinner appointment.

Figure 2: A web app that books a dinner appointment with Grandma.

For that, we will need widgets for displaying an image, a toggle button, a date picker, an integer slider, and a button. We will go through this tutorial for all the widgets but not for the layout. Feel free to refer to the given Git repo to see the complete notebook.

First, import the ipywidgets module and the IPython display function:

import ipywidgets as widgets
from IPython.display import display, clear_output

The code for all widgets follows a similar and consistent format. Take the code for a toggle button as an example:

grand = widgets.ToggleButtons(
            options=['grandson', 'granddaughter']
        )

The ToggleButtons function from the widgets module is called. Its options parameter is filled with some button names that we want. For this, two toggle buttons are created: grandson and granddaughter. This function is passed to a variable named grand.

The following examples show how straight-forward it is to create any widget:

• A text box

name = widgets.Text(placeholder='Your name here')

• A date picker

date = widgets.DatePicker(description='Pick a Date')

• An integer slider

friends = widgets.IntSlider(
            value=3, 
            min=0, 
            max=10,
            step=1,
            style={'description_width': 'initial' 
          )

We need to create an extra function to configure the actions to be performed when the button is pressed:

button_send = widgets.Button(
                description='Send to grandma',
                tooltip='Send',
                style={'description_width': 'initial'}
            )

output = widgets.Output()

def on_button_clicked(event):
    with output:
        clear_output()
        print("Sent message: ")
        print(f"Dear Grandma! This is your favourite {grand.value}, {name.value}.")
        print(f"I would love to come over on {date.value} for dinner, if that's okay for you!")
        print(f"Also, if you don't mind, I'll bring along {friends.value} hungry ghosts for your delicious food!")

button_send.on_click(on_button_clicked)

vbox_result = widgets.VBox([button_send, output])

Notice how literal it is to get the information from all widgets:

grand.value
name.value
date.value
friends.value

Done? Then we are ready to turn the notebook into a web application.

Step Three: Render the Notebook as A Web Application using Voilà

As we have installed the Voilà package, we can now see the button appeared on our notebook interface.

Figure 3: A Voilà button that renders a notebook as a web application.

There are two ways to debug the notebook as a web application before we host it somewhere else. Either press the Voilà button or execute the notebook from the terminal as follows:

$ voila notebook_name.ipynb

When any of them is executed, a new tab will appear on our default web browser. It shows how our web application looks like in the local directory. Take your time to edit and complete your web app.

Step Four: Push the Code Project onto Your Git Repository

Next, we need to create a requirements text file for the web app. To do that, type the following command in your activated virtual environment:

$ pip freeze > requirements.txt

The file includes all packages that you have installed in the current virtualenv. Remove the following lines from the file as they are incompatible with Binder:

pywin32==300
terminado==0.9.2

We are now ready to commit and push our project folder onto our Git repo by typing the following commands:

$ git add notebook.ipynb
$ git add requirements.txt
$ git commit -m “commit message”
$ git push origin branch_name

Notice the changes taking place on our Git repo.

Step Five: Host the Web Application on Binder

Now, we want the notebook to be displayed as a web app that is hosted on Binder, which anyone can access with a URL. Go to mybinder.org and choose the appropriate configurations for the code project.

Figure 4: A sample configuration for hosting a Git project on Binder.

Elaboration for Figure 4:

1. Choose your Git hosting platform, such as GitHub, GitLab, etc.
2. Insert the URL of your Git repo.
3. Insert the name of your notebook after '/voila/render/'.
4. Choose URL instead of File.
5. Click the Launch button.
6. Copy the text for displaying the Binder badge and paste it on the README file. Your web app will be shown when the badge is clicked on the Git repo.

And… voilà! Now you have a simple web application to showcase your work.

Conclusion

This article explains how to create a web application in a Jupyter notebook. It only needs basic Python programming knowledge. The ipywidgets module is used to create interactive widgets. The voilà module is used to render the notebook as a web application. The web app is hosted on Binder, which is accessible by a URL.

Programmer Humor

There are only 10 kinds of people in this world: those who know binary and those who don’t.

~~~

There are 10 types of people in the world. Those who understand trinary, those who don’t, and those who mistake it for binary.

Reference: [1] https://ipywidgets.readthedocs.io/en/stable/examples/Widget%20List.html

More Projects and Learnings on the Finxter Academy!

Do you want to keep doing real things with your coding skills rather than working on toy projects forever?

If yes, feel free to check out our full course on this subject — and many more similar coding projects for real value creation on the Finxter Academy: