v5.0.x: ofi: NIC selection update

[naughtont3]

The existing code in compare_cpusets assumed that some non_io ancestor of a PCI object should intersect with the cpuset of the proc. However, this is not true. There is a case where the non IO ancestor can be an L3. If there exists two L3s on the same NUMA and the process is bound to one L3, but the PCI object is connected to the other L3, then compare_cpusets() will return false.

A better way to determine the optimal interface is by finding the distances of the interfaces from the current process. Then find out which of these interfaces is nearest the process and select it.

Use the PMIx distance generation for this purpose.

Move away from using deprecated PMIX macros and use the functions directly instead.

Signed-off-by: Amir Shehata shehataa@ornl.gov
(cherry picked from commit d4e1ae5)

[wenduwan]

We are currently experiencing #11637 related to this change. Shall we hold onto it for a while before we come up with a fix?

[lrbison]

To clarify to what @wenduwan said, There are two problems.

1. main branch has some segfaults when using internal hwloc. @rhc54 and @amirshehataornl seem to be making good progress towards a solution in Do not compute distances when unbound #11641, but we should not merge this into 5.0.x without that change as well.

2. main branch currently cannot compile with external hwloc of 1.11.0, even though that is our stated minimum version. This affects Amazon Linux 2 platforms, and potentially others. We either need to agree to bump hwloc minimum version (I haven't checked what version this patch requires) or work around the version limitation.
   Edit: I see (2) should be fixed by Ralph's PR as well!

[jsquyres]

Also, #11657 needs to be resolved (and the fix added here) before this can be merged.

Converted this PR to "Draft" to ensure that it is not merged prematurely.

[wenduwan]

This change is also needed which fixes multiple bugs #11641

[naughtont3]

See revised PR #11739