btl: ofi remove FI_MR_PROV_KEY

[naughtont3]

No description provided.

[wenduwan]

Running AWS CI

[hppritcha]

@janjust could someone from nvidia check out what's going on with nvidia ci?

[hppritcha]

@wenduwan someone should check if this change is going to work with EFA provider

[wenduwan]

This change broke AWS CI. I need to understand the root cause.

[hppritcha]

i think this PR is incorrect for several providers.

[hppritcha]

@BrendanCunningham could you check if this works with OPX provider?

[BrendanCunningham]

@BrendanCunningham could you check if this works with OPX provider?

@tmh97 Can you take a look at this?

@hppritcha I don't work on OPX myself; I maintain PSM2. Thomas Huber works on OPX.

[wenduwan]

I figured the problem with EFA provider is that it requries FI_MR_PROV_KEY in mr_mode. Therefore this change effectively disabled EFA in btl/ofi.

That said, it's not clear to me how to get what you want. I need to ask the libfabric community.

[wenduwan]

@amirshehataornl @naughtont3 Can you attest to ofiwg/libfabric#9777 ? Is that what you want?

[j-xiong]

What is the intention of this change? Does the ofi btl now always wants to supply the mr keys instead of letting the provider choose the key? If it can work either way the mode bit should remain. Providers that don't require the mode will clear the flag and MPI should check the flag returned from fi_getinfo() to determine how the keys should be supplied.

[hppritcha]

i see no evidence this is needed by the CXI provider and since it breaks support for EFA not sure we could consider merging this, certainly not as is.

[wenduwan]

Closing PR due to inactivity. Please reopen if you still need it.