May-29-2018, 09:43 AM
Hi,
I had a python script which executes different processes in parallel.
How can I modify this script so that it can run the parallel processes in a pool of 5 at a time?
Please help
I had a python script which executes different processes in parallel.
How can I modify this script so that it can run the parallel processes in a pool of 5 at a time?
Please help
from multiprocessing import Pool from multiprocessing import Process import glob import os import csv from threading import Thread from multiprocessing import Pool as ThreadPool import time import multiprocessing patterns = ['ACM','ACX','AW','BC','XU0', 'DRM', 'DHD', 'CR', 'GSK', 'DMS', 'BLS'] def process(pattern=[]): base_path = '/ai2/data/dev/admin/inf/*{}*' print("ID of process: {}".format(os.getpid())) print "Name of the process is :" ,pattern search_path = base_path.format(pattern) for f in glob.glob(search_path): print("-----------------------") print ("The directory path is:") print f print("List of files in the directory are:") os.chdir('/ai2/data/dev/admin/inf/') os.chdir(f) cwd = os.getcwd() for subdir, dirs, files in os.walk(cwd, topdown=True): for file23 in glob.glob('*.err'): print file23 if __name__ == "__main__": # confirms that the code is under main function patterns = ['ACM','ACX','AW','BC','XU0', 'DRM', 'DHD', 'CR', 'GSK', 'DMS', 'BLS'] procs = [] #proc = Process(target=process) # instantiating without any argument #procs.append(proc) #proc.start() # instantiating process with arguments for pattern in patterns: #print(name) proc = Process(target=process,args=(pattern,)) procs.append(proc) proc.start() Output:ID of process: 5789 Name of the process is : ACM ----------------------- The directory path is: /ai2/data/dev/admin/inf/inf_ACM_pvt List of files in the directory are: dsplit.err ID of process: 5793 Name of the process is : XU0 ----------------------- The directory path is: /ai2/data/dev/admin/inf/inf_XU0_pvt List of files in the directory are: t_itm.err ID of process: 5791 Name of the process is : AW ----------------------- The directory path is: /ai2/data/dev/admin/inf/inf_AW_pvt List of files in the directory are: aware.err ID of process: 5796 Name of the process is : CR ----------------------- The directory path is: /ai2/data/dev/admin/inf/inf_CR_pvt List of files in the directory are: issue.err ID of process: 5792 Name of the process is : BC ----------------------- The directory path is: /ai2/data/dev/admin/inf/inf_BC_pvt List of files in the directory are: run_ingest_BC_daily_1249.err ID of process: 5797 Name of the process is : GSK ----------------------- The directory path is: /ai2/data/dev/admin/inf/inf_GSK_pvt List of files in the directory are: gs_attribute.err ID of process: 5790 Name of the process is : ACX ----------------------- The directory path is: /ai2/data/dev/admin/inf/inf_ACX_pvt List of files in the directory are: accountshighfocus.err acm_access_log.err ID of process: 5795 Name of the process is : DHD ----------------------- The directory path is: /ai2/data/dev/admin/inf/inf_DHD_pvt List of files in the directory are: retpaid.err ID of process: 5794 Name of the process is : DRM ----------------------- The directory path is: /ai2/data/dev/admin/inf/inf_DRM_pvt List of files in the directory are: call_report_y9clookup.err ID of process: 5798 Name of the process is : DMS ----------------------- The directory path is: /ai2/data/dev/admin/inf/inf_DMS_pvt List of files in the directory are: run_ingest_DMS_daily_1235.err ID of process: 5799 Name of the process is : BLS ----------------------- The directory path is: /ai2/data/dev/admin/inf/inf_BLS_pvt List of files in the directory are: run_ingest_BLS_daily_1241.err 