💡 Problem Formulation: Parallel execution in Python enables the running of multiple tasks concurrently, improving throughput and efficiency, particularly on multi-core systems. For example, processing large datasets or handling numerous I/O-bound tasks can be parallelised to decrease execution time significantly.
Method 1: Using the threading Module
The threading module in Python is used to run multiple threads (lighter-weight processes) concurrently. Threads share the same memory space, making it easy to share data between them, but care must be taken to avoid conflicts.
Here’s an example:
import threading
def print_numbers():
for i in range(5):
print(i)
thread1 = threading.Thread(target=print_numbers)
thread2 = threading.Thread(target=print_numbers)
thread1.start()
thread2.start()
thread1.join()
thread2.join()Output:
0
0
1
1
2
2
3
3
4
4
This code snippet creates two threads, both executing the print_numbers() function. The function prints numbers from 0 to 4. By calling start(), both threads begin their execution. The join() method ensures that the main program waits for both threads to complete before continuing.
Method 2: Using the multiprocessing Module
To utilize multiple processor cores for true parallelism, Python offers the multiprocessing module. Each process runs in its own Python interpreter, and communication between processes is achieved through IPC methods.
Here’s an example:
from multiprocessing import Process
def print_numbers():
for i in range(5):
print(i)
process1 = Process(target=print_numbers)
process2 = Process(target=print_numbers)
process1.start()
process2.start()
process1.join()
process2.join()Output:
0
0
1
1
2
2
3
3
4
4
This example creates two separate processes that run the print_numbers() function. Similar to threading, the start() and join() methods are used to control the execution flow, but processes do not share memory unlike threads.
Method 3: Using concurrent.futures
The concurrent.futures module provides a high-level interface for asynchronously executing callables. The ThreadPoolExecutor and ProcessPoolExecutor classes allow you to manage a pool of threads or processes for executing tasks in parallel.
Here’s an example:
from concurrent.futures import ThreadPoolExecutor
def print_numbers():
for i in range(5):
print(i)
with ThreadPoolExecutor(max_workers=2) as executor:
executor.submit(print_numbers)
executor.submit(print_numbers)Output:
0
1
2
3
4
0
1
2
3
4
This code uses ThreadPoolExecutor to create a pool of workers that executes two submitted tasks in parallel. The with statement is used to manage the pool, closing it once all tasks are completed.
Method 4: Using Joblib
Joblib is an external library specialized in pipelining Python jobs for computation and I/O operations. It is easy to use and can often lead to performance improvements with minimal code changes.
Here’s an example:
from joblib import Parallel, delayed
def print_numbers(i):
print(i)
Parallel(n_jobs=2)(delayed(print_numbers)(i) for i in range(5))Output:
0
1
2
3
4
The Parallel function creates a pool of workers, and delayed is a wrapper to capture the arguments of a function. This code runs print_numbers() for numbers 0 to 4 in parallel across two jobs.
Bonus One-Liner Method 5: Using List Comprehensions with multiprocessing
List comprehensions combined with the multiprocessing module provide a concise way to execute functions in parallel.
Here’s an example:
from multiprocessing import Pool
with Pool(2) as p:
print(p.map(print_numbers, range(5)))Output:
[None, None, None, None, None]
This one-liner uses a pool of workers to map the print_numbers() function over a range of numbers, executing the function in parallel. The print_numbers function would be modified to handle an input parameter.
Summary/Discussion
- Method 1: threading. Pros: Lightweight and easy to share data between threads. Cons: Not suitable for CPU-bound tasks due to Python’s Global Interpreter Lock (GIL).
- Method 2: multiprocessing. Pros: True parallel execution, circumventing the GIL. Cons: More overhead due to separate memory spaces and inter-process communication.
- Method 3: concurrent.futures. Pros: Simplified API for handling parallel tasks. Cons: Still subject to the limitations of Python threading or multiprocessing, depending on the executor used.
- Method 4: Joblib. Pros: Simplified pipelining and often improved performance. Cons: Dependency on an external library.
- Bonus One-Liner Method 5: List Comprehensions with multiprocessing. Pros: Conciseness and simplicity. Cons: Less control over individual task execution.