default maxpendtime set for 1 day

[shahzebsiddiqui]

This PR will enforce we have a default maxpendtime set for 1 day. This address bug in #1593 reported by @karcaw. I can confirm this error is present in codebase. I was able to reproduce this error when submitting a job. We can test this feature by holding the job permanently so add -H.

Here is an example on Perlmutter

buildspecs: hostname_perlmutter: description: run hostname on perlmutter or muller type: script executor: '(perlmutter|muller).slurm.(regular|debug|preempt)' tags: ["queues","jobs"] sbatch: ["-t 5", "-n 1", "-N 1", "-C gpu", "-A m3503", '-H'] run: hostname

When building this test it was polling every 30sec and will continue to poll till it reaches 1 day. Shown below is output of the run phase

Polling Jobs in 30 seconds  Pending and Suspended Jobs ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓ ┃ builder ┃ executor ┃ jobid ┃ jobstate ┃ runtime ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩ │ hostname_perlmutter/34101ea5 │ perlmutter.slurm.regular │ 14179008 │ PENDING │ 34.416 │ │ hostname_perlmutter/4f8f3be3 │ perlmutter.slurm.debug │ 14179007 │ PENDING │ 34.457 │ │ hostname_perlmutter/53570379 │ perlmutter.slurm.preempt │ 14179006 │ PENDING │ 34.499 │ └──────────────────────────────┴──────────────────────────┴──────────┴──────────┴─────────┘ Polling Jobs in 30 seconds  Pending and Suspended Jobs ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓ ┃ builder ┃ executor ┃ jobid ┃ jobstate ┃ runtime ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩ │ hostname_perlmutter/34101ea5 │ perlmutter.slurm.regular │ 14179008 │ PENDING │ 64.57 │ │ hostname_perlmutter/4f8f3be3 │ perlmutter.slurm.debug │ 14179007 │ PENDING │ 64.611 │ │ hostname_perlmutter/53570379 │ perlmutter.slurm.preempt │ 14179006 │ PENDING │ 64.653 │ └──────────────────────────────┴──────────────────────────┴──────────┴──────────┴─────────┘ Polling Jobs in 30 seconds  Pending and Suspended Jobs ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓ ┃ builder ┃ executor ┃ jobid ┃ jobstate ┃ runtime ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩ │ hostname_perlmutter/34101ea5 │ perlmutter.slurm.regular │ 14179008 │ PENDING │ 94.724 │ │ hostname_perlmutter/4f8f3be3 │ perlmutter.slurm.debug │ 14179007 │ PENDING │ 94.763 │ │ hostname_perlmutter/53570379 │ perlmutter.slurm.preempt │ 14179006 │ PENDING │ 94.803 │ └──────────────────────────────┴──────────────────────────┴──────────┴──────────┴─────────┘ Polling Jobs in 30 seconds  Pending and Suspended Jobs ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓ ┃ builder ┃ executor ┃ jobid ┃ jobstate ┃ runtime ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩ │ hostname_perlmutter/34101ea5 │ perlmutter.slurm.regular │ 14179008 │ PENDING │ 124.874 │ │ hostname_perlmutter/4f8f3be3 │ perlmutter.slurm.debug │ 14179007 │ PENDING │ 124.913 │ │ hostname_perlmutter/53570379 │ perlmutter.slurm.preempt │ 14179006 │ PENDING │ 124.953 │ └──────────────────────────────┴──────────────────────────┴──────────┴──────────┴─────────┘ Polling Jobs in 30 seconds  Pending and Suspended Jobs ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓ ┃ builder ┃ executor ┃ jobid ┃ jobstate ┃ runtime ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩ │ hostname_perlmutter/34101ea5 │ perlmutter.slurm.regular │ 14179008 │ PENDING │ 155.024 │ │ hostname_perlmutter/4f8f3be3 │ perlmutter.slurm.debug │ 14179007 │ PENDING │ 155.065 │ │ hostname_perlmutter/53570379 │ perlmutter.slurm.preempt │ 14179006 │ PENDING │ 155.119 │ └──────────────────────────────┴──────────────────────────┴──────────┴──────────┴─────────┘

When one specifies --maxpendtime on the command line this value is honored and has the highest precendence

(buildtest) siddiq90@login02> buildtest bd -b perlmutter_hostname.yml --maxpendtime=15 ╭───────────────────────────────────────────────────── buildtest summary ──────────────────────────────────────────────────────╮ │ │ │ User: siddiq90 │ │ Hostname: login02 │ │ Platform: Linux │ │ Current Time: 2023/08/21 07:37:07 │ │ buildtest path: /global/homes/s/siddiq90/gitrepos/buildtest/bin/buildtest │ │ buildtest version: 1.5 │ │ python path: /global/u1/s/siddiq90/.local/share/virtualenvs/buildtest-WqshQcL1/bin/python3 │ │ python version: 3.9.7 │ │ Configuration File: /global/u1/s/siddiq90/gitrepos/buildtest-nersc/config.yml │ │ Test Directory: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests │ │ Report File: /global/u1/s/siddiq90/gitrepos/buildtest/var/report.json │ │ Command: /global/homes/s/siddiq90/gitrepos/buildtest/bin/buildtest bd -b perlmutter_hostname.yml --maxpendtime=15 │ │ │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ ───────────────────────────────────────────────────────────────────────────────────────────────── Discovering Buildspecs ─────────────────────────────────────────────────────────────────────────────────────────────────  Discovered buildspecs ╔══════════════════════════════════════════════════════════════════════════════════════════╗ ║ buildspec ║ ╟──────────────────────────────────────────────────────────────────────────────────────────╢ ║ /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/queues/perlmutter_hostname.yml ║ ╚══════════════════════════════════════════════════════════════════════════════════════════╝ Total Discovered Buildspecs: 1 Total Excluded Buildspecs: 0 Detected Buildspecs after exclusion: 1 ─────────────────────────────────────────────────────────────────────────────────────────────────── Parsing Buildspecs ──────────────────────────────────────────────────────────────────────────────────────────────────── Valid Buildspecs: 1 Invalid Buildspecs: 0 /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/queues/perlmutter_hostname.yml: VALID Total builder objects created: 3  Builders by type=script ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ description ┃ buildspecs ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ hostname_perlmutter/6ea344a2 │ script │ perlmutter.slurm.regular │ None │ None │ None │ run hostname on perlmutter or muller │ /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/queues/perlmutter_hostn… │ ├──────────────────────────────┼────────┼──────────────────────────┼──────────┼───────┼───────┼──────────────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ hostname_perlmutter/ced0716d │ script │ perlmutter.slurm.debug │ None │ None │ None │ run hostname on perlmutter or muller │ /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/queues/perlmutter_hostn… │ ├──────────────────────────────┼────────┼──────────────────────────┼──────────┼───────┼───────┼──────────────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ hostname_perlmutter/2f840152 │ script │ perlmutter.slurm.preempt │ None │ None │ None │ run hostname on perlmutter or muller │ /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/queues/perlmutter_hostn… │ └──────────────────────────────┴────────┴──────────────────────────┴──────────┴───────┴───────┴──────────────────────────────────────┴────────────────────────────────────────────────────────────────────────────────────┘  Batch Job Builders ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ builder ┃ executor ┃ buildspecs ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ hostname_perlmutter/6ea344a2 │ perlmutter.slurm.regular │ /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/queues/perlmutter_hostname.yml │ ├──────────────────────────────┼──────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────┤ │ hostname_perlmutter/ced0716d │ perlmutter.slurm.debug │ /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/queues/perlmutter_hostname.yml │ ├──────────────────────────────┼──────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────┤ │ hostname_perlmutter/2f840152 │ perlmutter.slurm.preempt │ /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/queues/perlmutter_hostname.yml │ └──────────────────────────────┴──────────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────┘ ────────────────────────────────────────────────────────────────────────────────────────────────────── Building Test ────────────────────────────────────────────────────────────────────────────────────────────────────── hostname_perlmutter/6ea344a2: Creating test directory: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.regular/perlmutter_hostname/hostname_perlmutter/6ea344a2 hostname_perlmutter/6ea344a2: Creating the stage directory: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.regular/perlmutter_hostname/hostname_perlmutter/6ea344a2/stage hostname_perlmutter/6ea344a2: Writing build script: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.regular/perlmutter_hostname/hostname_perlmutter/6ea344a2/hostname_perlmutter_build.sh hostname_perlmutter/ced0716d: Creating test directory: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.debug/perlmutter_hostname/hostname_perlmutter/ced0716d hostname_perlmutter/ced0716d: Creating the stage directory: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.debug/perlmutter_hostname/hostname_perlmutter/ced0716d/stage hostname_perlmutter/ced0716d: Writing build script: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.debug/perlmutter_hostname/hostname_perlmutter/ced0716d/hostname_perlmutter_build.sh hostname_perlmutter/2f840152: Creating test directory: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.preempt/perlmutter_hostname/hostname_perlmutter/2f840152 hostname_perlmutter/2f840152: Creating the stage directory: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.preempt/perlmutter_hostname/hostname_perlmutter/2f840152/stage hostname_perlmutter/2f840152: Writing build script: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.preempt/perlmutter_hostname/hostname_perlmutter/2f840152/hostname_perlmutter_build.sh ────────────────────────────────────────────────────────────────────────────────────────────────────── Running Tests ────────────────────────────────────────────────────────────────────────────────────────────────────── Spawning 8 processes for processing builders ─────────────────────────────────────────────────────────────────────────────────────────────────────── Iteration 1 ─────────────────────────────────────────────────────────────────────────────────────────────────────── hostname_perlmutter/2f840152 does not have any dependencies adding test to queue hostname_perlmutter/6ea344a2 does not have any dependencies adding test to queue hostname_perlmutter/ced0716d does not have any dependencies adding test to queue In this iteration we are going to run the following tests: [hostname_perlmutter/2f840152, hostname_perlmutter/6ea344a2, hostname_perlmutter/ced0716d] hostname_perlmutter/ced0716d: Current Working Directory : /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.debug/perlmutter_hostname/hostname_perlmutter/ced0716d/stage hostname_perlmutter/2f840152: Current Working Directory : /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.preempt/perlmutter_hostname/hostname_perlmutter/2f840152/stage hostname_perlmutter/6ea344a2: Current Working Directory : /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.regular/perlmutter_hostname/hostname_perlmutter/6ea344a2/stage hostname_perlmutter/2f840152: Running Test via command: bash --norc --noprofile -eo pipefail hostname_perlmutter_build.sh hostname_perlmutter/ced0716d: Running Test via command: bash --norc --noprofile -eo pipefail hostname_perlmutter_build.sh hostname_perlmutter/6ea344a2: Running Test via command: bash --norc --noprofile -eo pipefail hostname_perlmutter_build.sh hostname_perlmutter/6ea344a2: JobID 14179165 dispatched to scheduler hostname_perlmutter/ced0716d: JobID 14179166 dispatched to scheduler hostname_perlmutter/2f840152: JobID 14179167 dispatched to scheduler Polling Jobs in 30 seconds hostname_perlmutter/ced0716d: Cancelling Job 14179166 because job exceeds max pend time of 15 sec with current pend time of 30.156 sec hostname_perlmutter/6ea344a2: Cancelling Job 14179165 because job exceeds max pend time of 15 sec with current pend time of 30.268 sec hostname_perlmutter/2f840152: Cancelling Job 14179167 because job exceeds max pend time of 15 sec with current pend time of 30.377 sec Unable to run any tests

Now i added maxpendtime in the configuration file this will be under defaults keyword shown be is a sample configuration on Perlmutter

 perlmutter: description: Cray Shasta system with AMD CPU and NVIDIA A100 GPUs hostnames: - login[01-40] moduletool: lmod poolsize: 8 buildspecs: rebuild: false count: 15 format: name,description terse: false report: count: 25 terse: false format: name,id,state,runtime,returncode latest: true oldest: false executors: defaults: pollinterval: 30 maxpendtime: 35

Now if i run this without specifying --maxpendtime it will take the value of the configuration file. Note we should have this test cancel at 60s since we poll every 30s and maxpendtime hasnt reached

(buildtest) siddiq90@login02> buildtest bd -b perlmutter_hostname.yml ╭───────────────────────────────────────────── buildtest summary ─────────────────────────────────────────────╮ │ │ │ User: siddiq90 │ │ Hostname: login02 │ │ Platform: Linux │ │ Current Time: 2023/08/21 07:39:35 │ │ buildtest path: /global/homes/s/siddiq90/gitrepos/buildtest/bin/buildtest │ │ buildtest version: 1.5 │ │ python path: /global/u1/s/siddiq90/.local/share/virtualenvs/buildtest-WqshQcL1/bin/python3 │ │ python version: 3.9.7 │ │ Configuration File: /global/u1/s/siddiq90/gitrepos/buildtest-nersc/config.yml │ │ Test Directory: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests │ │ Report File: /global/u1/s/siddiq90/gitrepos/buildtest/var/report.json │ │ Command: /global/homes/s/siddiq90/gitrepos/buildtest/bin/buildtest bd -b perlmutter_hostname.yml │ │ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ ───────────────────────────────────────────────────────────────────────────────────────────────── Discovering Buildspecs ───────────────────────────────────────────────────────────────────────────────────────────────── Discovered buildspecs ╔══════════════════════════════════════════════════════════════════════════════════════════╗ ║ buildspec ║ ╟──────────────────────────────────────────────────────────────────────────────────────────╢ ║ /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/queues/perlmutter_hostname.yml ║ ╚══════════════════════════════════════════════════════════════════════════════════════════╝ Total Discovered Buildspecs: 1 Total Excluded Buildspecs: 0 Detected Buildspecs after exclusion: 1 ─────────────────────────────────────────────────────────────────────────────────────────────────── Parsing Buildspecs ──────────────────────────────────────────────────────────────────────────────────────────────────── Valid Buildspecs: 1 Invalid Buildspecs: 0 /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/queues/perlmutter_hostname.yml: VALID Total builder objects created: 3 Builders by type=script ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ builder ┃ type ┃ executor ┃ compiler ┃ nodes ┃ procs ┃ description ┃ buildspecs ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ hostname_perlmutter/9eb614a3 │ script │ perlmutter.slurm.regular │ None │ None │ None │ run hostname on perlmutter or muller │ /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/queues/perlmutter_hostn… │ ├──────────────────────────────┼────────┼──────────────────────────┼──────────┼───────┼───────┼──────────────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ hostname_perlmutter/b1b8ecaf │ script │ perlmutter.slurm.debug │ None │ None │ None │ run hostname on perlmutter or muller │ /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/queues/perlmutter_hostn… │ ├──────────────────────────────┼────────┼──────────────────────────┼──────────┼───────┼───────┼──────────────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ hostname_perlmutter/f4547c2f │ script │ perlmutter.slurm.preempt │ None │ None │ None │ run hostname on perlmutter or muller │ /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/queues/perlmutter_hostn… │ └──────────────────────────────┴────────┴──────────────────────────┴──────────┴───────┴───────┴──────────────────────────────────────┴────────────────────────────────────────────────────────────────────────────────────┘ Batch Job Builders ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ builder ┃ executor ┃ buildspecs ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ hostname_perlmutter/9eb614a3 │ perlmutter.slurm.regular │ /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/queues/perlmutter_hostname.yml │ ├──────────────────────────────┼──────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────┤ │ hostname_perlmutter/b1b8ecaf │ perlmutter.slurm.debug │ /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/queues/perlmutter_hostname.yml │ ├──────────────────────────────┼──────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────┤ │ hostname_perlmutter/f4547c2f │ perlmutter.slurm.preempt │ /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/queues/perlmutter_hostname.yml │ └──────────────────────────────┴──────────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────┘ ────────────────────────────────────────────────────────────────────────────────────────────────────── Building Test ────────────────────────────────────────────────────────────────────────────────────────────────────── hostname_perlmutter/9eb614a3: Creating test directory: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.regular/perlmutter_hostname/hostname_perlmutter/9eb614a3 hostname_perlmutter/9eb614a3: Creating the stage directory: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.regular/perlmutter_hostname/hostname_perlmutter/9eb614a3/stage hostname_perlmutter/9eb614a3: Writing build script: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.regular/perlmutter_hostname/hostname_perlmutter/9eb614a3/hostname_perlmutter_build.sh hostname_perlmutter/b1b8ecaf: Creating test directory: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.debug/perlmutter_hostname/hostname_perlmutter/b1b8ecaf hostname_perlmutter/b1b8ecaf: Creating the stage directory: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.debug/perlmutter_hostname/hostname_perlmutter/b1b8ecaf/stage hostname_perlmutter/b1b8ecaf: Writing build script: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.debug/perlmutter_hostname/hostname_perlmutter/b1b8ecaf/hostname_perlmutter_build.sh hostname_perlmutter/f4547c2f: Creating test directory: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.preempt/perlmutter_hostname/hostname_perlmutter/f4547c2f hostname_perlmutter/f4547c2f: Creating the stage directory: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.preempt/perlmutter_hostname/hostname_perlmutter/f4547c2f/stage hostname_perlmutter/f4547c2f: Writing build script: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.preempt/perlmutter_hostname/hostname_perlmutter/f4547c2f/hostname_perlmutter_build.sh ────────────────────────────────────────────────────────────────────────────────────────────────────── Running Tests ────────────────────────────────────────────────────────────────────────────────────────────────────── Spawning 8 processes for processing builders ─────────────────────────────────────────────────────────────────────────────────────────────────────── Iteration 1 ─────────────────────────────────────────────────────────────────────────────────────────────────────── hostname_perlmutter/9eb614a3 does not have any dependencies adding test to queue hostname_perlmutter/f4547c2f does not have any dependencies adding test to queue hostname_perlmutter/b1b8ecaf does not have any dependencies adding test to queue In this iteration we are going to run the following tests: [hostname_perlmutter/9eb614a3, hostname_perlmutter/f4547c2f, hostname_perlmutter/b1b8ecaf] hostname_perlmutter/b1b8ecaf: Current Working Directory : /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.debug/perlmutter_hostname/hostname_perlmutter/b1b8ecaf/stage hostname_perlmutter/9eb614a3: Current Working Directory : /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.regular/perlmutter_hostname/hostname_perlmutter/9eb614a3/stage hostname_perlmutter/f4547c2f: Current Working Directory : /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.preempt/perlmutter_hostname/hostname_perlmutter/f4547c2f/stage hostname_perlmutter/f4547c2f: Running Test via command: bash --norc --noprofile -eo pipefail hostname_perlmutter_build.sh hostname_perlmutter/b1b8ecaf: Running Test via command: bash --norc --noprofile -eo pipefail hostname_perlmutter_build.sh hostname_perlmutter/9eb614a3: Running Test via command: bash --norc --noprofile -eo pipefail hostname_perlmutter_build.sh hostname_perlmutter/f4547c2f: JobID 14179217 dispatched to scheduler hostname_perlmutter/9eb614a3: JobID 14179218 dispatched to scheduler hostname_perlmutter/b1b8ecaf: JobID 14179219 dispatched to scheduler Polling Jobs in 30 seconds Pending and Suspended Jobs ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓ ┃ builder ┃ executor ┃ jobid ┃ jobstate ┃ runtime ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩ │ hostname_perlmutter/f4547c2f │ perlmutter.slurm.preempt │ 14179217 │ PENDING │ 30.155 │ │ hostname_perlmutter/b1b8ecaf │ perlmutter.slurm.debug │ 14179219 │ PENDING │ 30.192 │ │ hostname_perlmutter/9eb614a3 │ perlmutter.slurm.regular │ 14179218 │ PENDING │ 30.231 │ └──────────────────────────────┴──────────────────────────┴──────────┴──────────┴─────────┘ Polling Jobs in 30 seconds hostname_perlmutter/f4547c2f: Cancelling Job 14179217 because job exceeds max pend time of 35 sec with current pend time of 60.307 sec hostname_perlmutter/b1b8ecaf: Cancelling Job 14179219 because job exceeds max pend time of 35 sec with current pend time of 61.653 sec hostname_perlmutter/9eb614a3: Cancelling Job 14179218 because job exceeds max pend time of 35 sec with current pend time of 63.043 sec Unable to run any tests