Multiprocessing

What is Multiprocessing?

Multiprocessing uses separate processes (not threads) for true parallelism. Each process has its own Python interpreter, bypassing the GIL.

Basic Multiprocessing

Use the multiprocessing module:

import multiprocessing
import time

def worker(name):
    print(f"Worker {name} starting")
    time.sleep(2)
    print(f"Worker {name} finished")

if __name__ == "__main__":
    processes = []
    for i in range(3):
        p = multiprocessing.Process(target=worker, args=(i,))
        processes.append(p)
        p.start()
    
    for p in processes:
        p.join()

ProcessPoolExecutor

Easier process management:

from concurrent.futures import ProcessPoolExecutor

def square(x):
    return x ** 2

numbers = [1, 2, 3, 4, 5]

with ProcessPoolExecutor() as executor:
    results = executor.map(square, numbers)
    print(list(results))  # [1, 4, 9, 16, 25]

Sharing Data

Use shared memory or queues:

import multiprocessing

def worker(queue):
    queue.put("Hello from worker")

if __name__ == "__main__":
    queue = multiprocessing.Queue()
    p = multiprocessing.Process(target=worker, args=(queue,))
    p.start()
    p.join()
    print(queue.get())

When to Use

• CPU-bound tasks: Mathematical computations
• True parallelism: Need multiple CPU cores
• Avoid for I/O-bound: Use threading or async instead

Best Practices

• Use if __name__ == "__main__": Required on Windows
• Use ProcessPoolExecutor: Easier than manual processes
• Consider overhead: Processes have more overhead than threads
• Use queues: For inter-process communication

Next Steps

Learn about multithreading and async/await.