sometimes CI stuck because test_recognize_digits run forever #8168
Description
The loop termination condition (below some loss threshold) could never reach because the model reached a bad state with NaN loss:
[22:37:57] : [Step 1/1] 72/72 Test #351: test_recognize_digits ...................***Exception: Other2588.00 sec [22:37:57] : [Step 1/1] PassID 0, BatchID 0010, Test Loss 1.7, Acc 0.41 [22:37:57] : [Step 1/1] PassID 0, BatchID 0020, Test Loss 1.1, Acc 0.71 [22:37:57] : [Step 1/1] PassID 0, BatchID 0030, Test Loss 0.68, Acc 0.8 [22:37:57] : [Step 1/1] PassID 0, BatchID 0040, Test Loss 0.51, Acc 0.84 [22:37:57] : [Step 1/1] infer results: [[0.04630879 0.00167811 0.12129738 0.34336442 0.14744693 0.03341632 [22:37:57] : [Step 1/1] 0.01416888 0.00522202 0.27912304 0.00797407]] [22:37:57] : [Step 1/1] .PassID 0, BatchID 0010, Test Loss 1.4, Acc 0.59 [22:37:57] : [Step 1/1] PassID 0, BatchID 0020, Test Loss 0.86, Acc 0.72 [22:37:57] : [Step 1/1] PassID 0, BatchID 0030, Test Loss 0.59, Acc 0.83 [22:37:57] : [Step 1/1] .EEPassID 0, BatchID 0010, Test Loss 1.3, Acc 0.62 [22:37:57] : [Step 1/1] PassID 0, BatchID 0020, Test Loss 0.8, Acc 0.75 [22:37:57] : [Step 1/1] PassID 0, BatchID 0030, Test Loss 0.65, Acc 0.8 [22:37:57] : [Step 1/1] PassID 0, BatchID 0040, Test Loss 0.54, Acc 0.84 [22:37:57] : [Step 1/1] infer results: [[0.21086314 0.12968424 0.05096722 0.27543962 0.03747584 0.00835655 [22:37:57] : [Step 1/1] 0.09035194 0.00980698 0.14980537 0.03724904]] [22:37:57] : [Step 1/1] .PassID 0, BatchID 0010, Test Loss 1.2, Acc 0.65 [22:37:57] : [Step 1/1] PassID 0, BatchID 0020, Test Loss 0.77, Acc 0.78 [22:37:57] : [Step 1/1] PassID 0, BatchID 0030, Test Loss 0.57, Acc 0.85 [22:37:57] : [Step 1/1] .PassID 0, BatchID 0010, Test Loss 1.3, Acc 0.65 [22:37:57] : [Step 1/1] PassID 0, BatchID 0020, Test Loss 0.79, Acc 0.76 [22:37:57] : [Step 1/1] PassID 0, BatchID 0030, Test Loss 0.63, Acc 0.82 [22:37:57] : [Step 1/1] PassID 0, BatchID 0040, Test Loss 0.52, Acc 0.85 [22:37:57] : [Step 1/1] PassID 0, BatchID 0050, Test Loss 0.51, Acc 0.84 [22:37:57] : [Step 1/1] .PassID 0, BatchID 0010, Test Loss nan, Acc 0.028 [22:37:57] : [Step 1/1] PassID 0, BatchID 0020, Test Loss nan, Acc 0.028 [22:37:57] : [Step 1/1] PassID 0, BatchID 0030, Test Loss nan, Acc 0.028 [22:37:57] : [Step 1/1] PassID 0, BatchID 0040, Test Loss nan, Acc 0.028 [22:37:57] : [Step 1/1] PassID 0, BatchID 0050, Test Loss nan, Acc 0.028 [22:37:57] : [Step 1/1] PassID 0, BatchID 0060, Test Loss nan, Acc 0.028 [22:37:57] : [Step 1/1] PassID 0, BatchID 0070, Test Loss nan, Acc 0.028 [22:37:57] : [Step 1/1] PassID 0, BatchID 0080, Test Loss nan, Acc 0.028