Python Video to Text – Speech Recognition

by

A good friend and his wife recently founded an AI startup in the lifestyle niche that uses machine learning to discover specific real-world patterns from videos.

For their business system, they need a pipeline that takes a video file, converts it to audio, and transcribes the audio to standard text that is then used for further processing. I couldn’t help but work on a basic solution to help fix their business problem.

Project Overview

I finished the project in three steps:

  • First, install the necessary libraries.
  • Second, convert the video to an audio file (.mp4 to .wav)
  • Third, convert the audio file to a speech file (.wav to .txt). We first break the large audio file into smaller chunks and convert each of them separately due to the size restrictions of the used API.

Let’s get started!

Step 1: Install Libraries

We need the following import statements in our code:

# Import libraries
import speech_recognition as sr
import os
from pydub import AudioSegment
from pydub.silence import split_on_silence
import moviepy.editor as mp

Consequently, you need to pip install the following three libraries in your shell — assuming you run Python version 3.9:

pip3.9 install pydub
pip3.9 install SpeechRecognition
pip3.9 install moviepy

The os module is already preinstalled as a Python Standard Library.

If you need an additional guide on how to install Python libraries, check out this tutorial:

👉 Recommended: Python Install Library Guide

Step 2: Video to Audio

Before you can do speech recognition on the video, we need to extract the audio as a .wav file using the moviepy.editor.VideoFileClip().audio.write_audiofile() method.

Here’s the code:

def video_to_audio(in_path, out_path):
    """Convert video file to audio file"""
    
    video = mp.VideoFileClip(in_path)
    video.audio.write_audiofile(out_path)

👉 Recommended: Python Video to Audio

Step 3: Audio to Text

After extracting the audio file, we can start transcribing the speech from the .wav file using Google’s powerful speech recognition library on chunks of the potentially large audio file.

Using chunks instead of passing the whole audio file avoids an error for large audio files — Google has some restrictions on the audio file size.

However, you can play around with the splitting thresholds of 700ms silence—it can be more or less, depending on your concrete file.

Here’s the audio to text code function that worked for me:

def large_audio_to_text(path):
    """Split audio into chunks and apply speech recognition"""
    
    # Open audio file with pydub
    sound = AudioSegment.from_wav(path)

    # Split audio where silence is 700ms or greater and get chunks
    chunks = split_on_silence(sound, min_silence_len=700, silence_thresh=sound.dBFS-14, keep_silence=700)
    
    # Create folder to store audio chunks
    folder_name = "audio-chunks"
    if not os.path.isdir(folder_name):
        os.mkdir(folder_name)
    
    whole_text = ""
    # Process each chunk
    for i, audio_chunk in enumerate(chunks, start=1):
        # Export chunk and save in folder
        chunk_filename = os.path.join(folder_name, f"chunk{i}.wav")
        audio_chunk.export(chunk_filename, format="wav")

        # Recognize chunk
        with sr.AudioFile(chunk_filename) as source:
            audio_listened = r.record(source)
            # Convert to text
            try:
                text = r.recognize_google(audio_listened)
            except sr.UnknownValueError as e:
                print("Error:", str(e))
            else:
                text = f"{text.capitalize()}. "
                print(chunk_filename, ":", text)
                whole_text += text

    # Return text for all chunks
    return whole_text

Need more info? Check out the following deep dive:

👉 Recommended: Large Audio to Text? Here’s My Speech Recognition Solution in Python

Step 4: Putting It Together

Finally, we can combine our functions. First, we extract the audio from the video. Second, we chunk the audio into smaller files and recognize speech independently on each chunk using Google’s speech recognition module.

I added comments to annotate the most important parts of this code:

# Import libraries
import speech_recognition as sr
import os
from pydub import AudioSegment
from pydub.silence import split_on_silence
import moviepy.editor as mp


def video_to_audio(in_path, out_path):
    """Convert video file to audio file"""
    
    video = mp.VideoFileClip(in_path)
    video.audio.write_audiofile(out_path)

    
def large_audio_to_text(path):
    """Split audio into chunks and apply speech recognition"""
    
    # Open audio file with pydub
    sound = AudioSegment.from_wav(path)

    # Split audio where silence is 700ms or greater and get chunks
    chunks = split_on_silence(sound, min_silence_len=700, silence_thresh=sound.dBFS-14, keep_silence=700)
    
    # Create folder to store audio chunks
    folder_name = "audio-chunks"
    if not os.path.isdir(folder_name):
        os.mkdir(folder_name)
    
    whole_text = ""
    # Process each chunk
    for i, audio_chunk in enumerate(chunks, start=1):
        # Export chunk and save in folder
        chunk_filename = os.path.join(folder_name, f"chunk{i}.wav")
        audio_chunk.export(chunk_filename, format="wav")

        # Recognize chunk
        with sr.AudioFile(chunk_filename) as source:
            audio_listened = r.record(source)
            # Convert to text
            try:
                text = r.recognize_google(audio_listened)
            except sr.UnknownValueError as e:
                print("Error:", str(e))
            else:
                text = f"{text.capitalize()}. "
                print(chunk_filename, ":", text)
                whole_text += text

    # Return text for all chunks
    return whole_text


# Create a speech recognition object
r = sr.Recognizer()

# Video to audio to text
video_to_audio('sample_video.mp4', 'sample_audio.wav')
result = large_audio_to_text('sample_audio.wav')

# Print to shell and file
print(result)
print(result, file=open('result.txt', 'w'))

Store this code in a folder next to your video file 'sample_video.mp4' and run it. It will create an audio file 'sample_audio.wav' and chunk the audio and print the result to the shell, as well as to a file called 'result.txt'. This contains the transcription of the video file.