← retour aux snippets

ProcessPoolExecutor: CPU-bound et chunksize

Paralléliser des calculs CPU avec map et chunksize adapté.

python concurrency #processpool#cpu

objectif

Paralléliser des calculs CPU avec map et chunksize adapté.

code minimal

from concurrent.futures import ProcessPoolExecutor
def square(x): return x*x
with ProcessPoolExecutor() as ex:
    res = list(ex.map(square, range(5), chunksize=2))
print(res == [0,1,4,9,16])  # attendu: True

utilisation

from concurrent.futures import ProcessPoolExecutor
def f(x): return x+1
with ProcessPoolExecutor() as ex:
    print(sum(ex.map(f, [1,2,3])) == 9)

variante(s) utile(s)

from concurrent.futures import ProcessPoolExecutor
def id_(x): return x
with ProcessPoolExecutor() as ex:
    out = list(ex.map(id_, [], chunksize=1))
print(out == [])

notes

  • ProcessPool contourne le GIL pour CPU-bound.
  • Ajustez chunksize pour réduire la surcharge IPC.