[WIP] Fix weights initialization of several vision models #19449
Conversation
NielsRogge force-pushed the vit_init_weights branch from 129f7b2 to a280d8c Compare 
| The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. |
amyeroberts left a comment
amyeroberts left a comment There was a problem hiding this comment.
Looking good! Just a few comments on updating the TF models and initialization calls in inits.
There was a problem hiding this comment.
How come it's only the VIT model that has these tests skipped?
There was a problem hiding this comment.
Can you make the equivalent initialization updates in the TF model?
There was a problem hiding this comment.
Can you make the equivalent initialization updates in the TF model?
There was a problem hiding this comment.
Can you make the equivalent updates for the TF model?
There was a problem hiding this comment.
I think you might need the post_init() call here
| self.config = config | |
| self.config = config | |
| # Initialize weights and apply final processing | |
| self.post_init() |
There was a problem hiding this comment.
And possibly here
| self.num_patches = num_patches | |
| # Initialize weights and apply final processing | |
| self.post_init() |
There was a problem hiding this comment.
I'd move this up out of the for-loop as it doesn't need to be redefined for each parameter
| This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Please note that issues that do not follow the contributing guidelines are likely to be ignored. |
NielsRogge added the WIP NielsRogge force-pushed the vit_init_weights branch from b298da1 to be67d33 Compare 
| Any further updates? @NielsRogge @amyeroberts |
5 participants
What does this PR do?
This PR is a follow-up of #19341, to make sure weights are properly initialized when training vision models from scratch.