5 Best Ways to Loop Through Directories and Subdirectories in Python

💡 Problem Formulation: When working with file systems in Python, it’s common to need to iterate over all the files in a directory and its subdirectories. This could be for tasks such as searching for a specific file, summarizing content, or batch processing. Input could be a root directory path, and the desired output is an iterative process that traverses through that directory and all its children, accessing each file and perhaps printing its path.

Method 1: Using os.walk()

The os.walk() is a generator that yields a tuple of 3 values (dirpath, dirnames, filenames) for each directory in the tree, including the root directory. It is a simple and widely-used method to traverse directory trees in Python. This function is part of Python’s standard library, which makes it very accessible and easy to use.

Here’s an example:

import os

root_path = '/path/to/directory'
for dirpath, dirnames, filenames in os.walk(root_path):
    for filename in filenames:
        print(os.path.join(dirpath, filename))

Output:

/path/to/directory/file1.txt
/path/to/directory/subdirectory/file2.txt
...

This code iterates over the directories and subdirectories starting from the specified root_path. Within the inner loop, it prints out the full path to every file by joining the dirpath with each filename.

Method 2: Using pathlib.Path.rglob()

The pathlib module provides object-oriented filesystem paths and the Path.rglob(pattern) method, which allows for recursive globbing (pattern matching) in directories. It is preferred for its readability and the use of the intuitive Path objects over strings.

Here’s an example:

from pathlib import Path

root_path = Path('/path/to/directory')
for path in root_path.rglob('*'):
    if path.is_file():
        print(path)

Output:

/path/to/directory/file1.txt
/path/to/directory/subdirectory/file2.txt
...

With rglob('*'), we match all files and directories under the given root_path. The if path.is_file(): condition filters out directories, allowing the print statement to output only file paths.

Method 3: Using glob.glob() with Recursive Option

glob.glob() returns a list of pathnames matching a specified pattern. By using the recursive wildcard pattern ‘**’, glob can be used for traversing directories recursively. However, you need to enable this feature by setting the recursive argument to True.

Here’s an example:

import glob

for file_path in glob.glob('/path/to/directory/**/*', recursive=True):
    if os.path.isfile(file_path):
        print(file_path)

Output:

/path/to/directory/file1.txt
/path/to/directory/subdirectory/file2.txt
...

This code uses a wildcard pattern to match all file paths recursively. The if os.path.isfile(file_path): check ensures that only files are printed, as glob returns directories as well.

Method 4: Using scandir() and Recursion

The os.scandir() function returns directory entries along with file attribute information, which can be used to avoid calling os.stat() multiple times. To traverse subdirectories, a recursive function which calls itself for each directory encountered is used.

Here’s an example:

import os

def scandir_recursive(directory):
    with os.scandir(directory) as entries:
        for entry in entries:
            if entry.is_file():
                print(entry.path)
            elif entry.is_dir():
                scandir_recursive(entry.path)

root_path = '/path/to/directory'
scandir_recursive(root_path)

Output:

/path/to/directory/file1.txt
/path/to/directory/subdirectory/file2.txt
...

The scandir_recursive function prints every file’s path it encounters and calls itself when it finds a directory. This approach is efficient due to the reduced number of system calls compared to other methods.

Bonus One-Liner Method 5: List Comprehension with os.walk()

For a quick and concise traversal, Python’s list comprehension can be used in conjunction with os.walk() to generate a list of file paths in a single line of code. It’s a compact yet powerful method for developers familiar with list comprehensions.

Here’s an example:

import os

file_paths = [os.path.join(dp, f) for dp, dn, filenames in os.walk('/path/to/directory') for f in filenames]
print("\n".join(file_paths))

Output:

/path/to/directory/file1.txt
/path/to/directory/subdirectory/file2.txt
...

This one-liner uses list comprehension to loop through the output of os.walk(), combining dp (directory path) with each filename f to generate the full path to each file in the directory tree.

Summary/Discussion

• Method 1: os.walk(). It is straightforward and doesn’t require any external libraries. However, it can be slow for very large directories.
• Method 2: pathlib.Path.rglob(). This method makes the code more readable and aligns with modern Python practices. It might be slightly slower than os.walk().
• Method 3: glob.glob() with Recursive. It is less efficient than os.walk() as it builds a list of all paths before iteration. This method can consume more memory for large directory trees.
• Method 4: scandir() and Recursion. This is a more efficient approach when file attributes are also needed. It is Python 3.5+ only.
• Bonus Method 5: List Comprehension with os.walk(). This method is quick to code but can be less efficient in terms of memory as it stores all paths in a list before printing them.