Skip to content

Memory leak in multiprocessing pool #28

@drew6017

Description

@drew6017

(windows10 / python3.6 / conda:latest / all-other-packages:latest)

There is a memory leak somewhere in the multiprocessing section:

with multiprocessing.Pool(n_worker) as pool:

Running 8 workers on a system with 16GB of ram will lead to a crash within ~1-2mins and a nasty pagefile increase to 40GB! I re-wrote the script using the older Process() api and no shared variables between instances which fixed the issue so its probably something to do with data passing.

All of my tests were done on the full celebahq dataset.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions