5 Best Ways to Traverse a Directory Recursively in Python

Rate this post

πŸ’‘ Problem Formulation: When working with file systems, a common task is to traverse directories recursively to handle files and folders. In Python, several methods exist to achieve this, each with its own use-case. Whether you’re summarizing directory contents, searching for files, or performing batch operations, the goal is to efficiently list all files and directories within a given path. For example, given a directory /my_folder, we want to recursively list all files and subdirectories contained within it.

Method 1: Using os.walk()

The os.walk() function is a versatile tool for traversing directory trees. It generates the file names in a directory tree by walking either top-down or bottom-up. For each directory within the tree, it yields a 3-tuple containing the directory path, directory names, and file names.

Here’s an example:

import os

for root, dirs, files in os.walk('/my_folder'):
    print(f'Current Path: {root}')
    print(f'Subdirectories: {dirs}')
    print(f'Files: {files}')
    print('--------------')

Output:

Current Path: /my_folder
Subdirectories: ['subfolder1', 'subfolder2']
Files: ['file1.txt', 'file2.txt']
--------------
Current Path: /my_folder/subfolder1
Subdirectories: []
Files: ['file3.txt']
--------------
... and so on for each subdirectory ...

This code uses os.walk() to iterate over the directory ‘/my_folder’. In each iteration, it prints the current directory path, the subdirectories within it, and the files contained. This method is comprehensive but may be overkill if one simply needs a list of files.

Method 2: Using Pathlib

The modern approach in Python 3.4+ uses the pathlib module, which provides an object-oriented interface for file system paths. Specifically, the Path.rglob() method is useful for recursive directory traversal, matching all files and directories with a specific pattern.

Here’s an example:

from pathlib import Path

for path in Path('/my_folder').rglob('*'):
    print(path.name)

Output:

file1.txt
file2.txt
file3.txt
... and so on for each file and subdirectory ...

The above code snippet leverages Path.rglob() to traverse all files and directories within ‘/my_folder’. rglob('*') matches everything, making it a simple and elegant way to list all items recursively.

Method 3: Using glob.glob() with the recursive flag

The glob module’s glob() function supports a recursive parameter as of Python 3.5+. This allows pattern matching for files in a directory recursively using the ‘**’ wildcard.

Here’s an example:

import glob

for filepath in glob.glob('/my_folder/**/*', recursive=True):
    print(filepath)

Output:

/my_folder/file1.txt
/my_folder/file2.txt
/my_folder/subfolder1/file3.txt
... and so on through the file system ...

The code uses the glob.glob() function with recursive=True to find all paths in ‘/my_folder’ and prints each one. This is convenient for filename pattern matching.

Method 4: Using os.listdir() in a Recursive Function

For a custom recursive directory traversal, combine os.listdir() with a recursive function. This can be useful for more complex traversal scenarios beyond just listing files and directories.

Here’s an example:

import os

def list_dir_recursive(directory):
    for entry in os.listdir(directory):
        full_path = os.path.join(directory, entry)
        if os.path.isdir(full_path):
            list_dir_recursive(full_path)
        else:
            print(full_path)

list_dir_recursive('/my_folder')

Output:

/my_folder/file1.txt
/my_folder/file2.txt
/my_folder/subfolder1/file3.txt
... and so on for each file ...

The function list_dir_recursive() is defined to print each file path within ‘/my_folder’, calling itself when encountering a directory. This is a more manual but highly customizable method.

Bonus One-Liner Method 5: Using os.scandir() and a Generator Expression

With Python 3.5+, os.scandir() can be used for more efficient directory iteration. Combined with a generator expression, one can create a compact recursive directory walker.

Here’s an example:

import os

def scandir_recursive(directory):
    for entry in os.scandir(directory):
        if entry.is_dir(follow_symlinks=False):
            yield from scandir_recursive(entry.path)
        else:
            yield entry.path

print(list(scandir_recursive('/my_folder')))

Output:

['/my_folder/file1.txt', '/my_folder/file2.txt', '/my_folder/subfolder1/file3.txt', ...]

The generator function scandir_recursive() is a Pythonic and efficient way to list all files in ‘/my_folder’, yielding file paths for each file encountered, recursively traversing subdirectories.

Summary/Discussion

  • Method 1: os.walk(). It’s the go-to method for many due to its simplicity. However, it can be slower than other methods, especially for large directory trees.
  • Method 2: Pathlib. Offers an elegant and OOP-based approach to file system paths. It’s a modern and readable technique but may not perform as well as os.scandir() for very large directory structures.
  • Method 3: glob.glob(). Ideal for pattern matching and concise code, but lacks the finer control and performance of more direct filesystem traversal methods.
  • Method 4: os.listdir() in a Recursive Function. It allows for tailor-made traversal logic, offering flexibility at the cost of more boilerplate code. A good choice for specific use-cases where control is crucial.
  • Bonus One-Liner Method 5: os.scandir(). Provides a minimalist yet efficient solution, useful when performance is a concern, though it lacks the simplicity of os.walk() at first glance.