Recently I have been thinking of a way to explicitly setup different environmental variables for different MPI processes. My case is to profile my own parallel CUDA code (using nsys) on my server. I am not quite sure which part this question belongs to - probably more to the shell environment and MPI mechanism? So please forgive me if this is posted to a wrong catalog...
Anyway, let's say I have a multi-process, multi-card code - i.e. one MPI proc using one MIG instance for calculation. And I use MPI for parallelization. The CUDA itself (or nsys) is not closely relevant to the question -
The tricky part is, I want to explicitly assign a certain MIG instance to a process using the system variable $CUDA_VISIBLE_DEVICES (because for MIG cards from Nvidia, somehow only the first card in the device list can be used).
An ideal setup would be something like:
mpirun -np 8 myscript.sh where for the "i"th process, it has its own environmental variable, like:
#! /bin/bash CUDA_VISIBLE_DEVICES=i nsys profile -o mylog$$ -f true -s none --cpuctxsw=none mycode so that the first process "spawned" from MPI gets the first card, the second gets the second card, etc. So the question will be:
is there a way to explicitly assign different environment variables for different shell scripts? (or is it possible at all?)
I understand that I can use $$ to distinguish between the processes, but surely each process (shell script here) does not know about any information of the other processes and wouldn't know the "rank" of itself, because we have not yet come to the MPI code when setting the variables. Is there a mechanism (from shell or MPI) that allows us to do that?
Thanks a lot in advance,
Hiroshi
