Python Forum
How to run different processes in a pool of 5
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
How to run different processes in a pool of 5
#1
Hi,

I had a python script which executes different processes in parallel.
How can I modify this script so that it can run the parallel processes in a pool of 5 at a time?
Please help
from multiprocessing import Pool from multiprocessing import Process import glob import os import csv from threading import Thread from multiprocessing import Pool as ThreadPool import time import multiprocessing patterns = ['ACM','ACX','AW','BC','XU0', 'DRM', 'DHD', 'CR', 'GSK', 'DMS', 'BLS'] def process(pattern=[]):	base_path = '/ai2/data/dev/admin/inf/*{}*'	print("ID of process: {}".format(os.getpid()))	print "Name of the process is :" ,pattern	search_path = base_path.format(pattern)	for f in glob.glob(search_path):	print("-----------------------")	print ("The directory path is:")	print f	print("List of files in the directory are:")	os.chdir('/ai2/data/dev/admin/inf/')	os.chdir(f)	cwd = os.getcwd()	for subdir, dirs, files in os.walk(cwd, topdown=True):	for file23 in glob.glob('*.err'):	print file23 if __name__ == "__main__": # confirms that the code is under main function	patterns = ['ACM','ACX','AW','BC','XU0', 'DRM', 'DHD', 'CR', 'GSK', 'DMS', 'BLS']	procs = []	#proc = Process(target=process) # instantiating without any argument	#procs.append(proc)	#proc.start() # instantiating process with arguments	for pattern in patterns:	#print(name)	proc = Process(target=process,args=(pattern,))	procs.append(proc)	proc.start() 
Output:
ID of process: 5789 Name of the process is : ACM ----------------------- The directory path is: /ai2/data/dev/admin/inf/inf_ACM_pvt List of files in the directory are: dsplit.err ID of process: 5793 Name of the process is : XU0 ----------------------- The directory path is: /ai2/data/dev/admin/inf/inf_XU0_pvt List of files in the directory are: t_itm.err ID of process: 5791 Name of the process is : AW ----------------------- The directory path is: /ai2/data/dev/admin/inf/inf_AW_pvt List of files in the directory are: aware.err ID of process: 5796 Name of the process is : CR ----------------------- The directory path is: /ai2/data/dev/admin/inf/inf_CR_pvt List of files in the directory are: issue.err ID of process: 5792 Name of the process is : BC ----------------------- The directory path is: /ai2/data/dev/admin/inf/inf_BC_pvt List of files in the directory are: run_ingest_BC_daily_1249.err ID of process: 5797 Name of the process is : GSK ----------------------- The directory path is: /ai2/data/dev/admin/inf/inf_GSK_pvt List of files in the directory are: gs_attribute.err ID of process: 5790 Name of the process is : ACX ----------------------- The directory path is: /ai2/data/dev/admin/inf/inf_ACX_pvt List of files in the directory are: accountshighfocus.err acm_access_log.err ID of process: 5795 Name of the process is : DHD ----------------------- The directory path is: /ai2/data/dev/admin/inf/inf_DHD_pvt List of files in the directory are: retpaid.err ID of process: 5794 Name of the process is : DRM ----------------------- The directory path is: /ai2/data/dev/admin/inf/inf_DRM_pvt List of files in the directory are: call_report_y9clookup.err ID of process: 5798 Name of the process is : DMS ----------------------- The directory path is: /ai2/data/dev/admin/inf/inf_DMS_pvt List of files in the directory are: run_ingest_DMS_daily_1235.err ID of process: 5799 Name of the process is : BLS ----------------------- The directory path is: /ai2/data/dev/admin/inf/inf_BLS_pvt List of files in the directory are: run_ingest_BLS_daily_1241.err
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Multiprocessing Pool Multiple Instances How to Kill by Pool ID sunny9495 0 1,902 Nov-16-2022, 05:57 AM
Last Post: sunny9495
  processes shall be parallel flash77 4 2,617 Sep-20-2022, 11:46 AM
Last Post: DeaD_EyE
  Sharing imported modules with Sub Processes? Stubblemonster 2 3,116 May-02-2022, 06:42 AM
Last Post: Stubblemonster
  Killing processes via python Lavina 2 5,885 Aug-04-2021, 06:20 AM
Last Post: warnerarc
  Pool multiprocessing - know current status in loop? korenron 0 2,474 Jul-28-2021, 08:49 AM
Last Post: korenron
  pool mysql error - not catch by try\except? korenron 1 3,483 Jul-05-2021, 11:26 AM
Last Post: ibreeden
  Python - Import file sequence into Media Pool jensenni 1 3,716 Feb-02-2021, 05:11 PM
Last Post: buran
  sharing variables between two processes Kiyoshi767 1 3,122 Nov-07-2020, 04:00 AM
Last Post: ndc85430
  2 or more processes on the write end of the same pipe Skaperen 4 7,228 Sep-27-2020, 06:41 PM
Last Post: Skaperen
  Errors using --processes parameter sonhospa 3 3,997 Jul-01-2020, 02:24 PM
Last Post: sonhospa

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020
This forum uses Lukasz Tkacz MyBB addons.
Forum use Krzysztof "Supryk" Supryczynski addons.