0

I'm trying to have SGE run job array tasks concurrently based on the job shares parameter of qsub but it seems not to be working as expected. Is there a way to enable concurrent task execution based on shares?

I have a script which sleeps to simulate long running tasks and I submit it to a small SGE cluster (26 slots) as different job arrays as follows:

qsub -t 1-201 -js 100 sge_longRunning.sh qsub -t 1-202 -js 101 sge_longRunning.sh qsub -t 1-203 -js 102 sge_longRunning.sh

I would expect the tasks to be almost equally distributed on the cluster over time but what I get is that the last submitted array gets fully executed (all 203 tasks), then the second one gets completely executed, finally the first one.

The cluster operates under a functional policy with 1M tickets and 0.9 weight for functional policy tickets.

Any hints how to get the tasks for the different job arrays to run concurrently sharing almost equally the available resources? Any hint what might be wrong with the above configuration/test setting?

Improve this question

1 Answer 1

Reset to default
0

About the only practical way would be to submit the jobs as different user or project with its own share.

If that isn't practical then try submitting as one big array job which chooses which work to do based on a queue maintained by your script in whatever order you like.

Improve this answer
3
  • So do you confirm that the -js parameter of qsub is not working as expected? I thought it could also be a matter of too short / sleeping tasks but I could not develop a real workload test for it... Commented Jul 5, 2019 at 12:39
  • It tweaks the share of your tickets assigned to each job not the share of the cluster assigned to each job. Commented Jul 7, 2019 at 13:14
  • Since my user is the only one using the cluster and there are no projects I would then expect that tweaking the share of my tickets and the share of the cluster would produce the same result, namely parallel and not sequential job/task processing. Commented Jul 8, 2019 at 15:04

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.