btl/uct: reduce number of messages sent when establishing connections #13018
Conversation
The btl/uct code can be quite aggressive at sends connection messages over the connection endpoint. This could lead to a large number of unnecessary messages in some cases. This commit adds code to restrict the retry rate to 2ms. This timing is controlled by a new MCA variable: btl_uct_connection_retry_timeout. Signed-off-by: Nathan Hjelm <hjelmn@google.com>
| is there a simple test which demonstrates the problem that this PR is addressing? |
| UD is a bit different on our hardware and the extra message don't break things per-se but I added logging of UD sends and, for a 384 (2 * 192 ppn) process run, there were over 250,000 UD messages in a mostly connected case when there should have been no more than 2 * 192^2 (each process should send one for each off-node connection). With this fix the number dropped to 20k because it was not fully connecting. I may actually axe the retry entirely since UD and other similar TLs are treated as reliable by UCT. I have another change coming in to clean up the code a bit (leaving the retry) which has been fully tested but I need some time to test without the retry mechanism. |
hppritcha left a comment
hppritcha left a comment There was a problem hiding this comment.
i can't really test this but built for me and passed basic multi-node sanity using ob1.
2 participants
The btl/uct code can be quite aggressive at sends connection messages over the connection endpoint. This could lead to a large number of unnecessary messages in some cases. This commit adds code to restrict the retry rate to 2ms. This timing is controlled by a new MCA variable: btl_uct_connection_retry_timeout.