Parallel execution

The parallel verb family runs a callable over a batch of items – on threads, processes, or asyncio – with a progress bar. The how-to guide (Run a batch in parallel with progress) shows the idioms; this page is the API reference.

The shared keywords

Every verb accepts (where applicable):

Keyword	What it does
workers	Pool size for the sync verbs. Default: the executor’s own default (min(32, cpus + 4) threads, cpus processes). Accepted as an alias for concurrency on the async verbs.
concurrency	Async verbs: maximum in-flight tasks. None (default) creates every task up front, like asyncio.gather.
pool	'thread' (default), 'process', 'interpreter' (Python 3.14+), or an existing concurrent.futures.Executor instance (used as-is, never shut down for you).
bar	'plain' (one aggregate bar, default), 'multi' (a MultiBar with one sub-bar per in-flight task), False (no output), or a configured ProgressBar/MultiBar instance to drive.
on_error	'raise' (default): first failure cancels pending work and re-raises. 'return': exceptions appear in place of their results; KeyboardInterrupt/SystemExit still propagate.
chunksize	Items per task. Default: 1 on threads; automatic on process and interpreter pools (about 16 chunks per worker, capped at 1000). The bar advances per chunk.
buffersize	Maximum unfinished submitted tasks (sync verbs). Default max(4 × workers, 16); keeps memory flat on huge or lazy inputs.
timeout	Overall deadline in seconds. Expiry cancels pending work and raises TimeoutError without waiting for running tasks.
poll_interval	Seconds between coordinator wakeups and no-progress bar redraws (default 0.1) – one knob for both.
initializer, initargs, mp_context, max_tasks_per_child, thread_name_prefix	Forwarded verbatim to the executor constructor (max_tasks_per_child needs Python 3.11+; each option is validated against the pool kind).
**bar_kwargs	Anything else goes to the bar: prefix=/desc=, suffix=, widgets=, max_value=, … Unknown names raise TypeError.

Sync verbs

progressbar.map(fn: Callable[[...], Any], /, *iterables: Iterable[Any], **kwargs: Any) → list[Any]

Apply fn to every zipped item in parallel; results in order.

The parallel counterpart of the builtin map: progressbar.map(fn, items, workers=8) runs on a thread pool by default, renders a progress bar, and returns the results in input order once the batch completes. pool='process' switches to processes, bar='multi' shows per-task sub-bars, and on_error='return' swaps fail-fast for exceptions-in-place. See execute for the full keyword reference.

progressbar.imap(fn: Callable[[...], Any], /, *iterables: Iterable[Any], **kwargs: Any) → Generator[Any, None, None]

Lazily apply fn in parallel, yielding results in input order.

The parallel counterpart of multiprocessing.Pool.imap: same ordering, same laziness, same results-only element shape. Results completed out of order are held back until their turn; the held set stays bounded by the submission window (buffersize).

Closing the generator early (break) cancels unsubmitted work and shuts down the run’s executor; wrap in contextlib.closing for deterministic cleanup. See execute for keywords.

progressbar.imap_unordered(fn: Callable[[...], Any], /, *iterables: Iterable[Any], **kwargs: Any) → Generator[tuple[Any, Any], None, None]

Lazily apply fn in parallel, yielding as tasks complete.

Yields (item, result) pairs in completion order – the pair shape exists because completion order loses the input correspondence (a deliberate deviation from multiprocessing.Pool.imap_unordered, which yields bare results). With multiple iterables, item is the argument tuple.

Closing the generator early cancels unsubmitted work; see imap.

progressbar.starmap(fn: Callable[[...], Any], iterable: Iterable[Any], /, **kwargs: Any) → list[Any]

map over pre-tupled arguments (multiprocessing.Pool.starmap).

starmap(fn, [(1, 2), (3, 4)]) calls fn(1, 2) and fn(3, 4) in parallel. See execute for keywords.

progressbar.thread_map(fn: Callable[[...], Any], /, *iterables: Iterable[Any], **kwargs: Any) → list[Any]

map pinned to a thread pool (tqdm-compatible spelling).

progressbar.process_map(fn: Callable[[...], Any], /, *iterables: Iterable[Any], **kwargs: Any) → list[Any]

map pinned to a process pool (tqdm-compatible spelling).

progressbar.as_completed(futures: Iterable[Future[Any]], timeout: float | None = None, *, bar: Any = 'plain', poll_interval: float = 0.1, **bar_kwargs: Any) → Generator[Future[Any], None, None]

concurrent.futures.as_completed with a progress bar.

A superset of the stdlib function: same yield order and timeout semantics, plus a bar counting completions (total inferred from the futures). The caller owns the futures – an early break or a timeout never cancels them.

progressbar.run(command: str | Sequence[str] | Callable[[Any], Sequence[str]], items: Iterable[Any], /, *, check: bool = True, capture_output: bool = True, text: bool = True, shell: bool = False, cwd: Any = None, env: Any = None, **kwargs: Any) → list[CompletedProcess[Any]]

Run a shell command for every item in parallel, with a bar.

progressbar.run('gzip -k {}', files, workers=4) is a progress-bar’d xargs -P. Subprocesses release the GIL, so this always runs on threads (Pool.run reuses a pool’s executor).

Parameters:

• command – Template – see build_argv for the three forms and the placeholder rules.

• items – The batch; each becomes one subprocess.

• check – Raise subprocess.CalledProcessError on a non-zero exit (feeding on_error like any other worker error).

• capture_output – Capture stdout/stderr into the results – the default, so child output cannot corrupt the bar.

• text – Decode captured output as text.

• shell – Run through the shell (str form only). The items are substituted into the command line: only use with trusted items, this is the documented injection risk.

• cwd – Working directory for the subprocesses.

• env – Environment for the subprocesses.

• **kwargs – The shared execution keywords (workers, bar, on_error, timeout, …); see _sync.execute.

Returns:

One subprocess.CompletedProcess per item, in input order (exceptions in place under on_error='return').

Async verbs

async progressbar.amap(fn: Callable[[...], Any], /, *iterables: Iterable[Any], **kwargs: Any) → list[Any]

Apply fn to every zipped item on the event loop; ordered.

The async counterpart of progressbar.map. fn may be an async or a plain sync callable – sync callables run in a thread via asyncio.to_thread. Results come back in input order:

results = await progressbar.amap(fetch, urls, concurrency=8)

See execute_async for the keyword reference.

async progressbar.aimap(fn: Callable[[...], Any], /, *iterables: Iterable[Any], **kwargs: Any) → AsyncIterator[Any]

Lazily apply fn on the event loop, yielding in input order.

The async counterpart of imap: results-only, ordered, with out-of-order completions held back until their turn. Use contextlib.aclosing for deterministic cleanup on early exit. See execute_async for keywords.

async progressbar.aimap_unordered(fn: Callable[[...], Any], /, *iterables: Iterable[Any], **kwargs: Any) → AsyncIterator[tuple[Any, Any]]

Lazily apply fn on the event loop, yielding as tasks finish.

The async counterpart of imap_unordered: (item, result) pairs in completion order (the pair shape restores the correspondence completion order loses). See execute_async for keywords.

async progressbar.gather(*awaitables: Awaitable[Any], return_exceptions: bool = False, bar: Any = 'plain', poll_interval: float = 0.1, timeout: float | None = None, **bar_kwargs: Any) → list[Any]

asyncio.gather with a progress bar.

A drop-in replacement: results in argument order, no arguments yields [], and return_exceptions keeps asyncio’s exact keyword (mapped to on_error='return' internally). Unlike amap there is no concurrency limiting – the awaitables already exist, matching asyncio.gather semantics.

Reusable layers

class progressbar.Pool(workers: int | None = None, kind: str = 'thread', *, executor: Executor | None = None, **defaults: Any)

A reusable executor plus per-call defaults for the sync verbs.

The flat verbs create and destroy an executor per call; a Pool keeps one alive across calls:

with progressbar.Pool(8) as pool:
    first = pool.map(fetch, urls)
    second = pool.map(fetch, more_urls)

Positional shorthand: Pool(8) is eight threads, Pool(8, 'process') eight processes. Pool(executor=existing) adopts a caller-owned executor (never shut down here). Every other keyword becomes a per-call default that individual calls can override.

The executor is created lazily on first use, so an unused Pool(kind='process') spawns nothing.

Validate eagerly (fail fast); create nothing yet.

property executor: Executor

The underlying executor, created on first access.

map(fn: Callable[[...], Any], /, *iterables: Iterable[Any], **kwargs: Any) → list[Any]

map on this pool’s executor; see the module map.

imap(fn: Callable[[...], Any], /, *iterables: Iterable[Any], **kwargs: Any) → Generator[Any, None, None]

imap on this pool’s executor; see the module imap.

imap_unordered(fn: Callable[[...], Any], /, *iterables: Iterable[Any], **kwargs: Any) → Generator[tuple[Any, Any], None, None]

imap_unordered on this pool’s executor; see the module verb.

starmap(fn: Callable[[...], Any], iterable: Iterable[Any], /, **kwargs: Any) → list[Any]

starmap on this pool’s executor; see the module starmap.

run(command: Any, items: Iterable[Any], /, **kwargs: Any) → list[Any]

run a shell command per item on this pool’s executor.

shutdown(wait: bool = True, *, cancel_futures: bool = False) → None

Shut down the owned executor; adopted executors are spared.

class progressbar.AsyncPool(concurrency: int | None = None, **defaults: Any)

Shared concurrency limit plus per-call defaults for async verbs.

The async sibling of Pool. There is no executor to manage – tasks run on the caller’s event loop – so this is configuration reuse: a concurrency bound and default keywords applied to every call, overridable per call:

async with progressbar.AsyncPool(8) as pool:
    first = await pool.map(fetch, urls)
    async for item, result in pool.imap_unordered(fetch, more):
        ...

Store the concurrency bound and per-call defaults.

map(fn: Callable[[...], Any], /, *iterables: Iterable[Any], **kwargs: Any) → Coroutine[Any, Any, list[Any]]

amap with this pool’s limit and defaults; awaitable.

imap(fn: Callable[[...], Any], /, *iterables: Iterable[Any], **kwargs: Any) → AsyncIterator[Any]

aimap with this pool’s limit and defaults.

imap_unordered(fn: Callable[[...], Any], /, *iterables: Iterable[Any], **kwargs: Any) → AsyncIterator[tuple[Any, Any]]

aimap_unordered with this pool’s limit and defaults.

progressbar.parallel(**config: Any) → Callable[[Callable[[...], Any]], ParallelFunction]

Attach the parallel batch verbs to a function.

Parameters:

**config – Default keywords for every attached verb (workers, pool, bar, on_error, …); individual calls override them.

Returns:

A decorator returning the same function object with .map, .imap, .imap_unordered, .starmap, .amap, .aimap and .aimap_unordered attached.

Raises:

TypeError – The target is not a plain named function. Lambdas and bound methods are rejected because pool='process' pickles the function by qualified name.

progressbar.current_task_bar() → ProgressBar | None

Return the calling task’s own progress bar, if it has one.

Inside a function executed by progressbar.map/amap with bar='multi' this returns the per-task bar so the worker can report sub-progress (current_task_bar().update(i)). Anywhere else – including process-pool workers, which cannot share a bar object with the parent in v1 – it returns None.

class progressbar.ParallelFunction(*args, **kwargs)

A function enriched with the parallel batch verbs.

aimap(*iterables: Any, **kwargs: Any) → AsyncIterator[Any]

Async lazy ordered results; see the module verb.

aimap_unordered(*iterables: Any, **kwargs: Any) → AsyncIterator[tuple[Any, Any]]

Async completion-order pairs; see the module verb.

amap(*iterables: Any, **kwargs: Any) → Coroutine[Any, Any, list[Any]]

Async ordered map; see the module verb.

imap(*iterables: Any, **kwargs: Any) → Generator[Any, None, None]

Lazy ordered results; see the module verb.

imap_unordered(*iterables: Any, **kwargs: Any) → Generator[tuple[Any, Any], None, None]

Completion-order pairs; see the module verb.

map(*iterables: Any, **kwargs: Any) → list[Any]

Parallel ordered map over the iterables; see the module verb.

starmap(iterable: Any, **kwargs: Any) → list[Any]

Parallel map over pre-tupled arguments; see the module verb.