Fix None grad problem during training TOOD by adding SigmoidGeometricMean #7090
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Johnson-Wang changed the title 
ZwwWayne approved these changes Jan 28, 2022
ZwwWayne reviewed Jan 28, 2022
| from torch.nn import functional as F | ||
| | ||
| | ||
| class SigmoidGeometricMean(Function): |
There was a problem hiding this comment.
How about we implement an interface named sigmoid_geometric_mean = SigmoidGeometricMean.apply here so that in tood_head we can simply use sigmoid_geometric_mean(xxx)?
RangiLyu approved these changes Jan 28, 2022
Codecov Report
@@ Coverage Diff @@ ## dev #7090 +/- ## ========================================== + Coverage 62.41% 62.46% +0.04% ========================================== Files 330 330 Lines 26199 26216 +17 Branches 4436 4437 +1 ========================================== + Hits 16353 16375 +22 + Misses 8976 8966 -10 - Partials 870 875 +5
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
ZwwWayne approved these changes Jan 29, 2022
chhluo pushed a commit to chhluo/mmdetection that referenced this pull request Feb 21, 2022
ZwwWayne pushed a commit that referenced this pull request Jul 18, 2022
ZwwWayne pushed a commit to ZwwWayne/mmdetection that referenced this pull request Jul 19, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.
Motivation
The training of TOOD often encounters None gradient during backpropagation, which would further cause None tensors in the next training step. Some issues in the original repo (fcjian/TOOD#11) might be also due to this error. The problem is caused by the naive implementation of sigmoid geometric mean function
cls_score = (cls_logits.sigmoid() * cls_prob.sigmoid()).sqrt(). This output might be 0 ifcls_logitsorcls_probis a low negative value, which causes either inf grad of none grad during backpropagation.Modification
A reimplementation of
SigmoidGeometricMeanclass as an inheritance oftorch.autograd.Functionis proposed. The backward function is derived analytically and would avoid and inf or none grad during bp.Checklist