Skip to content

[muP] Rework #1087

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
lintangsutawika wants to merge 109 commits into
base: main
Choose a base branch
from
Open

[muP] Rework #1087

lintangsutawika wants to merge 109 commits into from

Conversation

@lintangsutawika
Copy link
Contributor

@lintangsutawika lintangsutawika commented Dec 1, 2023

No description provided.

lintangsutawika and others added 30 commits December 1, 2023 03:02
@lintangsutawika
changed ordering for setting up norm_factor
0d921f7
Update NeoXArgs docs automatically
abee54d
@lintangsutawika
updated muP args to the minimum required
a08c3ef
@lintangsutawika
calculate m_width
c35e830
@lintangsutawika
Merge branch 'main' of https://github.com/EleutherAI/gpt-neox into re…
2807e52 
…work-mup
@lintangsutawika
Merge branch 'rework-mup' of https://github.com/EleutherAI/gpt-neox i…
2d127df 
…nto rework-mup
Update NeoXArgs docs automatically
81fdc4d
@lintangsutawika
changed ordering for setting up norm_factor
7d6b246
@lintangsutawika
updated muP args to the minimum required
a0d1929
@lintangsutawika
calculate m_width
d63b3b8
@lintangsutawika
Update NeoXArgs docs automatically
9be82fe
@lintangsutawika
removed redundant line
66214d9
@lintangsutawika
removed redundant lines
17b7183
Update NeoXArgs docs automatically
a6bad07
@lintangsutawika
removed redundant lines
63984bd
@lintangsutawika
Merge branch 'rework-mup' of https://github.com/EleutherAI/gpt-neox i…
02687a8 
…nto rework-mup
Update NeoXArgs docs automatically
11114e2
@lintangsutawika
modify init with mup
05c4de3
@lintangsutawika
divide logits by the m_width
71a91e4
@lintangsutawika
moved position of mup parameters being processed
99c8ce0
@lintangsutawika
add note
b253ab6
@lintangsutawika
made param groups to hold flag for mup scaling
1919499
@lintangsutawika
lr scale
17678e0
@lintangsutawika
update config
2bd5ae6
@lintangsutawika
adjust process of mup variables
6642291
@lintangsutawika
remove calling save_base_shapes
8be6c66
@lintangsutawika
lr adjustments is done in train_step to address lr being reset due to…
c9fb18b 
… lr_scheduling
@lintangsutawika
lr scaling for mup is moved here instead
795371c
@lintangsutawika
removed mup usage for coord check
087beee
@lintangsutawika
merged with main
16d04b1
lintangsutawika and others added 27 commits February 29, 2024 14:03
@lintangsutawika
update config
fa142ff
@lintangsutawika
fixed how params are sorted
ad2336f
@lintangsutawika
update how seed is computed
fe73bc3
@lintangsutawika
update to follow pre-commit format
a3bd44c
@lintangsutawika
update from main
56b6c9b
@lintangsutawika
update
2365fd5
Update NeoXArgs docs automatically
e8639a0
@lintangsutawika
fix lr weighting
47e1438
@lintangsutawika
hard set to 1.0 if neox_args.use_mup is false
a064f9b
@Quentin-Anthony
Merge branch 'main' into rework-mup
b0da27a
Update NeoXArgs docs automatically
6fe55f4
@lintangsutawika
add new parameters
8bf8bcd
@lintangsutawika
add parameter checks
7f0b033
@lintangsutawika
updates to argument processing for mup
f802869
@lintangsutawika
add data save and descriptions being printed
cc71104
@lintangsutawika
update mup
c8feb39
@lintangsutawika
update seed
b6b3a02
@lintangsutawika
remove print text
847e892
@lintangsutawika
fixed kv
1b0027c
@lintangsutawika
update
055596f
@lintangsutawika
update dewcriptions being printed
fabb45b
@lintangsutawika
removed unused lines
5ccf693
@lintangsutawika
Merge branch 'rework-mup' of https://github.com/EleutherAI/gpt-neox i…
9dd583b 
…nto rework-mup
@lintangsutawika
Merge branch 'main' into rework-mup
6a8ad71
Update NeoXArgs docs automatically
485cad4
@Quentin-Anthony
Merge branch 'main' into rework-mup
c291906
@Quentin-Anthony
Merge branch 'main' into rework-mup
1ac9add
@Quentin-Anthony Quentin-Anthony self-assigned this Oct 8, 2024
@Quentin-Anthony Quentin-Anthony marked this pull request as ready for review October 8, 2024 23:00
@Quentin-Anthony Quentin-Anthony self-requested a review as a code owner October 8, 2024 23:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

6 participants

@lintangsutawika @ndey96 @rezaarefi @jthestness @Quentin-Anthony