https://stackoverflow.com/questions/18414020/memory-usage-keep-growing-with-pythons-multiprocessing-pool
To avoid memory leaks:
def parallel(func, args, nthreads):
from multiprocessing import Pool
from contextlib import closing
with closing( Pool(nthreads) ) as p:
rs = p.imap_unordered(func, args)
return [_ for _ in rs]