Skip to content
This repository was archived by the owner on Jul 7, 2023. It is now read-only.
Closed
Changes from 1 commit
Commits
Show all changes
963 commits
Select commit Hold shift + click to select a range
d53c77c
num shards * batch _Size
rllin-fathom Sep 23, 2019
ca8e147
pad before batch
kevincvlam Sep 23, 2019
5087580
print shapes
kevincvlam Sep 23, 2019
7b1662b
debug
kevincvlam Sep 23, 2019
3d9deac
debug
kevincvlam Sep 23, 2019
9d0b3c7
axis 0
kevincvlam Sep 23, 2019
208d674
debug_tfprint
kevincvlam Sep 23, 2019
b8cea98
before and after
kevincvlam Sep 23, 2019
0ce0402
test
etragas-fathom Sep 23, 2019
dc8a28f
track
etragas-fathom Sep 23, 2019
f654de4
fix batch size
kevincvlam Sep 23, 2019
e43b234
debug print
kevincvlam Sep 23, 2019
933ac3e
drop remainder
kevincvlam Sep 23, 2019
680db1c
try just batch size
kevincvlam Sep 23, 2019
465dda9
add inputs example, inputs chunk
kevincvlam Sep 23, 2019
f7fd369
num shards
kevincvlam Sep 23, 2019
bf48f13
params[batch_size
kevincvlam Sep 23, 2019
b805b55
no prints
kevincvlam Sep 23, 2019
154e5cf
Merge pull request #134 from medicode/hamilton-chunking
etragas-fathom Sep 23, 2019
ff857cf
train on gpu
kevincvlam Sep 24, 2019
bb7c413
use padded_batch
rllin-fathom Sep 24, 2019
7f68fff
Merge branch 'feature/model_packing' of github.com:medicode/tensor2te…
rllin-fathom Sep 24, 2019
91980f7
hardcode 8192
rllin-fathom Sep 24, 2019
3666d5a
batch size
rllin-fathom Sep 24, 2019
3c36d23
hard code
rllin-fathom Sep 24, 2019
197a82a
prints and clean up
rllin-fathom Sep 25, 2019
911c608
use batch size = tokens
rllin-fathom Sep 25, 2019
e3498ca
revert pad_for_tpu
rllin-fathom Sep 25, 2019
cb1cc08
num shards for gpu
rllin-fathom Sep 25, 2019
98fb030
packs per batch
rllin-fathom Sep 26, 2019
dd7ad80
pad for inputs_chunk_mask
rllin-fathom Sep 27, 2019
1a1783f
pad targets
rllin-fathom Sep 27, 2019
6b88f99
reshape targets in eval
rllin-fathom Sep 27, 2019
7459e64
reshape before eval
rllin-fathom Sep 27, 2019
a0fb7f6
fix reshape
rllin-fathom Sep 27, 2019
5cfaba8
revert use_resource and cast_grad changes for gpu
rllin-fathom Oct 1, 2019
440b278
typo
rllin-fathom Oct 1, 2019
b98dbd0
clarify comment
rllin-fathom Oct 1, 2019
f1425a6
Merge pull request #136 from medicode/feature/slow
rllin-fathom Oct 1, 2019
e1edc0e
Fix typo
kevincvlam Oct 1, 2019
184b2e9
fix another typo
kevincvlam Oct 1, 2019
e6bc1ac
Merge pull request #137 from medicode/fix/cast-grad-typo
kevincvlam Oct 1, 2019
c295c0c
pull in revert of use_resource and cast_grad
rllin-fathom Oct 2, 2019
6b8bb91
Revert "pull in revert of use_resource and cast_grad"
rllin-fathom Oct 2, 2019
728e72d
remove dataset padding
rllin-fathom Oct 3, 2019
63215f4
remove padding from packedconditional
rllin-fathom Oct 3, 2019
79ea139
batch instead of padded_batch
rllin-fathom Oct 3, 2019
47ed56c
padded_batch
rllin-fathom Oct 3, 2019
b6bd3a7
Revert "Merge pull request #137 from medicode/fix/cast-grad-typo"
rllin-fathom Oct 4, 2019
3c3468a
Revert "Merge pull request #136 from medicode/feature/slow"
rllin-fathom Oct 4, 2019
f89cfae
Revert "Merge pull request #131 from medicode/rllin-fathom-patch-1-1"
rllin-fathom Oct 4, 2019
f2168ca
Revert "Merge pull request #130 from medicode/feature/ingest_packing"
rllin-fathom Oct 4, 2019
1525870
Merge pull request #138 from medicode/revert/use_resource
cbockman Oct 4, 2019
2333a10
Merge branch 'master' into feature/model_packing
rllin-fathom Oct 7, 2019
19d49bf
clean up standardize shapes
rllin-fathom Oct 7, 2019
0886c0c
Merge branch 'feature/model_packing' of github.com:medicode/tensor2te…
rllin-fathom Oct 7, 2019
b13cadd
import
rllin-fathom Oct 7, 2019
f0f5112
clean up so we switch on packed or unpacked rather than tpu or non tpu
rllin-fathom Oct 7, 2019
ed6be80
clarify print
rllin-fathom Oct 7, 2019
671e369
clean up
rllin-fathom Oct 8, 2019
d34439f
comments
rllin-fathom Oct 8, 2019
10f1f51
confusing comment
rllin-fathom Oct 8, 2019
5b8ca72
cleaner fathom block
rllin-fathom Oct 8, 2019
f258a64
clean
rllin-fathom Oct 8, 2019
f4ec578
comment
rllin-fathom Oct 8, 2019
1f83095
better name
rllin-fathom Oct 8, 2019
072b385
reshape
rllin-fathom Oct 9, 2019
dd7728b
no reshape in eval
rllin-fathom Oct 9, 2019
c618909
reshape
rllin-fathom Oct 9, 2019
99c579e
reshape for batch
rllin-fathom Oct 10, 2019
1c9c5a1
undo shape fix
rllin-fathom Oct 10, 2019
33e84c4
allow first shape to be free
rllin-fathom Oct 10, 2019
5b307ad
remove reshape
rllin-fathom Oct 10, 2019
d906225
reshape
rllin-fathom Oct 10, 2019
2f51c74
decode problems
etragas-fathom Oct 10, 2019
fc99e59
fix padder
etragas-fathom Oct 10, 2019
c5876cb
comments
rllin-fathom Oct 11, 2019
c6a4170
Merge branch 'feature/model_packing' of github.com:medicode/tensor2te…
rllin-fathom Oct 11, 2019
925c9e7
packed problem ex
etragas-fathom Oct 11, 2019
a349920
Revert "Merge pull request #138 from medicode/revert/use_resource"
rllin-fathom Oct 14, 2019
2a8795d
print
rllin-fathom Oct 15, 2019
eb5cfaa
more print
rllin-fathom Oct 15, 2019
09da0cc
print
rllin-fathom Oct 15, 2019
b40538c
logits as dict logits
rllin-fathom Oct 15, 2019
8daaf06
reshape labels
rllin-fathom Oct 15, 2019
b24ec22
pull from feature/model_packing
rllin-fathom Oct 15, 2019
bdd7995
pad_to_next_chunk_length
rllin-fathom Oct 15, 2019
1f5b9aa
fix
rllin-fathom Oct 15, 2019
563ac97
fix
rllin-fathom Oct 15, 2019
57786bf
comment
rllin-fathom Oct 15, 2019
cf06989
fix
rllin-fathom Oct 15, 2019
0c36a58
fh logging
rllin-fathom Oct 15, 2019
ebaf0bc
Merge branch 'feature/merge_model_packing' into feature/semi_packed_p…
etragas-fathom Oct 16, 2019
3b9e4ea
cleanup
etragas-fathom Oct 16, 2019
bb512cb
Merge branch 'feature/merge_model_packing' into feature/semi_packed_p…
etragas-fathom Oct 16, 2019
5bcb743
model hparam
etragas-fathom Oct 16, 2019
02eb1db
cleanup
etragas-fathom Oct 16, 2019
66c67f3
cleanup
etragas-fathom Oct 16, 2019
f637312
newline
etragas-fathom Oct 16, 2019
674def2
Merge pull request #140 from medicode/feature/merge_model_packing
cbockman Oct 16, 2019
890e5fa
Merge branch 'master' into feature/model_packing
rllin-fathom Oct 16, 2019
297c658
use problem is packed
etragas-fathom Oct 16, 2019
6d95e3b
bad var name
etragas-fathom Oct 16, 2019
e7b4c62
make sure to grab full amount on tpu for eval
rllin-fathom Oct 18, 2019
de128d1
skip pad batch
rllin-fathom Oct 19, 2019
09a429f
fix indent
rllin-fathom Oct 19, 2019
3d9306a
new prob name
etragas-fathom Oct 22, 2019
35f2252
use dict
etragas-fathom Oct 22, 2019
ac25c67
Merge branch 'master' into feature/semi_packed_problem
etragas-fathom Oct 22, 2019
30b0aa9
note
etragas-fathom Oct 22, 2019
36c2361
clean up irrelevant line change
etragas-fathom Oct 22, 2019
969bc64
restore line
etragas-fathom Oct 22, 2019
bf93290
clean up
rllin-fathom Oct 22, 2019
1d46d1a
clean up
rllin-fathom Oct 22, 2019
e0ecf12
comment
rllin-fathom Oct 22, 2019
8507b3e
comments and fix
rllin-fathom Oct 23, 2019
740181e
Merge pull request #133 from medicode/feature/model_packing
rllin-fathom Oct 23, 2019
2b28276
note
etragas-fathom Oct 23, 2019
44fcb50
Merge branch 'master' into feature/semi_packed_problem
etragas-fathom Oct 23, 2019
055d5f4
Merge pull request #142 from medicode/feature/semi_packed_problem
etragas-fathom Oct 23, 2019
a0c3d4c
pass in checkpoint path
etragas-fathom Nov 1, 2019
8c7d284
use eval ckpt path
etragas-fathom Nov 1, 2019
6e884fb
dropprint
etragas-fathom Nov 1, 2019
301b266
docstring
etragas-fathom Nov 1, 2019
3e7f22e
Merge pull request #143 from medicode/experiment/tpu_eval
cbockman Nov 4, 2019
da0c506
add prints
etragas-fathom Nov 11, 2019
c990328
fix hparams
etragas-fathom Nov 11, 2019
ae13e85
cleanup
etragas-fathom Nov 13, 2019
081709d
add asana
etragas-fathom Nov 13, 2019
efeb0d8
Merge pull request #144 from medicode/dream/packed_ingest
cbockman Nov 13, 2019
6016382
Adding reasonable error message for the length assert in input_fn
keirn-fathom Nov 25, 2019
577ae61
rename
etragas-fathom Nov 25, 2019
5514d8e
log fix
etragas-fathom Nov 25, 2019
39dbf85
Remove debug line
keirn-fathom Nov 25, 2019
c6eb619
Merge pull request #145 from medicode/fix/rename_bert_max_length
cbockman Nov 25, 2019
9e1349e
Added new eval retry logic
branko-fathom Nov 26, 2019
e248131
Added logging
branko-fathom Nov 27, 2019
59e9ed5
Fixed checkpoint_path
branko-fathom Nov 27, 2019
358c30b
Fix path bug
branko-fathom Nov 27, 2019
bbbc1a1
Extra logs
branko-fathom Nov 27, 2019
0ccb829
Added mocker flag parsing
keirn-fathom Nov 27, 2019
2e2c673
Merge branch 'master' into feature/parse_mock_flags
keirn-fathom Nov 27, 2019
3d5cff4
Specify base_hparams
keirn-fathom Nov 27, 2019
26509b7
Merge branch 'feature/parse_mock_flags' of github.com:medicode/tensor…
keirn-fathom Nov 27, 2019
965f77e
Log mocking operation
keirn-fathom Nov 27, 2019
4859360
Adding better logging
keirn-fathom Nov 27, 2019
735e3b0
Merge branch 'master' into eval_tpu_training_checkpoints
branko-fathom Nov 27, 2019
74d4cef
Style fixes
keirn-fathom Nov 27, 2019
a4a3ba6
Syntax
keirn-fathom Nov 27, 2019
b25ddd5
Move mocker hijack into create_hparams
keirn-fathom Nov 28, 2019
ce3c3c9
Better assert
keirn-fathom Nov 28, 2019
b216536
Move retry logic from t2t to diseaseTools
branko-fathom Nov 28, 2019
9fe5262
Better error checking
keirn-fathom Nov 28, 2019
5cd3ee2
Whitespace
keirn-fathom Nov 28, 2019
f2a76ee
Whitespace
keirn-fathom Nov 28, 2019
f795a1a
Set up flags in packed mocker
keirn-fathom Nov 28, 2019
ea467ea
Remove setup
keirn-fathom Nov 28, 2019
96d513b
Remnove unneeded import
keirn-fathom Nov 28, 2019
f10ed22
Comments
keirn-fathom Nov 28, 2019
55504d0
import cleanup
branko-fathom Nov 28, 2019
b1174e4
Better commenting flag checking
keirn-fathom Nov 28, 2019
30f6e01
Moving fathom logic into diseaseTools
keirn-fathom Nov 28, 2019
68eb655
Style
keirn-fathom Nov 28, 2019
965c120
Better arg handling
keirn-fathom Nov 28, 2019
c38bff0
Spacing issue, python is mean
branko-fathom Dec 2, 2019
96d05f3
Added context to function
branko-fathom Dec 3, 2019
add6aa3
CR cleanup
branko-fathom Dec 3, 2019
43f79aa
Abstracting conditional application of batch settings
keirn-fathom Dec 3, 2019
bf24c44
Don't take unneeded args by name
keirn-fathom Dec 3, 2019
d1f13b2
Remove mocker logic from t2t
keirn-fathom Dec 3, 2019
7677b99
Naming variables
keirn-fathom Dec 3, 2019
3b7183f
slightly better docstring
keirn-fathom Dec 3, 2019
be14b20
Better wrapping for pad_to_next_chunk_length
keirn-fathom Dec 4, 2019
87ce12d
Update tensor2tensor/data_generators/problem.py
rllin-fathom Dec 4, 2019
c60d5ad
Update tensor2tensor/data_generators/problem.py
rllin-fathom Dec 4, 2019
9561b45
Update tensor2tensor/data_generators/problem.py
rllin-fathom Dec 4, 2019
d24eaf9
Update tensor2tensor/data_generators/problem.py
rllin-fathom Dec 4, 2019
cb28720
Update tensor2tensor/data_generators/problem.py
rllin-fathom Dec 4, 2019
30e6779
pass flags
rllin-fathom Dec 4, 2019
040b5a4
merge
rllin-fathom Dec 4, 2019
f93220a
fix merge
rllin-fathom Dec 4, 2019
268794b
Fix import typo
keirn-fathom Dec 4, 2019
f0b9b66
added import for modify if mocked
keirn-fathom Dec 4, 2019
a1f0417
Leftover code fixes
keirn-fathom Dec 4, 2019
ebb00b5
dataset reassign bug fix
keirn-fathom Dec 4, 2019
949cac0
Merge pull request #147 from medicode/eval_tpu_training_checkpoints
branko-fathom Dec 5, 2019
298f288
Merge branch 'master' into fix/packed_mocker_fixes
keirn-fathom Dec 6, 2019
1a8e47d
Merge pull request #149 from medicode/fix/packed_mocker_fixes
rllin-fathom Dec 6, 2019
7210bf0
Assign dataset to result of maybe_preapre_for_chunking
keirn-fathom Dec 6, 2019
26cad1b
Merge pull request #151 from medicode/fix/maybe_chunk
cbockman Dec 6, 2019
099d327
Fix/chunking and tpu (#152)
keirn-fathom Dec 10, 2019
9688f50
Fix trigger smoke test step.
fathom-kim Jan 10, 2020
1750d39
Merge pull request #154 from medicode/hotfix/ci-smoketest
cbockman Jan 10, 2020
295959f
change refs of gcr.io to us.gcr.io
fathom-dcmshi Jan 31, 2020
a3dd3cb
Merge pull request #155 from medicode/gcr.io-refs
fathom-dcmshi Feb 4, 2020
4ef2db0
Update to TensorFlow 1.15.2 (#156)
keirn-fathom Feb 25, 2020
981a65f
Rename inputs_chunk_mask (#157)
keirn-fathom Feb 27, 2020
5e08a40
Removed packed prediction problem
branko-fathom Feb 27, 2020
5acadda
Merge branch 'master' into remove_packed_prediction
branko-fathom Feb 27, 2020
c2af108
Debug
branko-fathom Feb 28, 2020
bf777cb
Revert "Debug"
branko-fathom Feb 28, 2020
953f836
Pass correct value to yield_single_examples
branko-fathom Feb 28, 2020
18ab1e3
added comment back
branko-fathom Feb 28, 2020
07dea51
Flip logic
branko-fathom Feb 28, 2020
e13ac4e
move is_hparams_packed
rllin-fathom Mar 4, 2020
49ce7d3
move
rllin-fathom Mar 4, 2020
7768099
bare import
rllin-fathom Mar 4, 2020
ec2b69a
Merge pull request #159 from medicode/return_to_single_ingest
cbockman Mar 5, 2020
077cce1
exp warmpup
rllin-fathom Mar 10, 2020
79a799e
log syntax
rllin-fathom Mar 10, 2020
6cb72bd
Merge pull request #160 from medicode/feature/lr_finder
rllin-fathom Mar 10, 2020
bdf6d21
Use same key for both authentication calls
kevincvlam Mar 10, 2020
6875905
Update config.yml
kevincvlam Mar 10, 2020
c5761a9
Merge pull request #161 from medicode/use-same-key
kevincvlam Mar 10, 2020
a222a25
new key
peiranguo Mar 11, 2020
9898e9e
Merge pull request #162 from medicode/buildcop/rotate-circleci-key
peiranguo Mar 11, 2020
2c37648
rotate sa keys
fathom-ragib Mar 18, 2020
54be654
Merge pull request #163 from medicode/ragib/rotate-sa-keys
cbockman Mar 18, 2020
79fbbb1
Fix return typing of tpu_batch_size_per_shard
alvinzgao Mar 19, 2020
457c546
Merge pull request #165 from medicode/fix/batch-size-typing
alvinzgao Mar 19, 2020
7f98f0e
Rename max_examples_per_pack to max_docs_per_packed_example
alvinzgao Mar 23, 2020
f4b130f
Merge pull request #166 from medicode/fix/rename-examples-per-pack
cbockman Mar 24, 2020
32d4821
CircleCI key rotation 2020-04-02
fathom-matthew Apr 2, 2020
fb925ea
Merge pull request #167 from medicode/circleci-key-rotation-2020-04-02
fathom-matthew Apr 3, 2020
254e0cd
Update config.yml (#169)
keirn-fathom Apr 8, 2020
34b6b7e
CircleCI key rotation: 2020-04-08 (2)
fathom-matthew Apr 9, 2020
777d96c
Merge pull request #171 from medicode/circleci-key-rotation-2020-04-0…
fathom-matthew Apr 9, 2020
a3d2ca5
Update config.yml
branko-fathom Apr 15, 2020
600eceb
Merge pull request #174 from medicode/branko-fathom-patch-1
branko-fathom Apr 15, 2020
e11cafc
CircleCI Service Account Key Rotation 2020-04-30 (#176)
fathom-matthew Apr 30, 2020
e938883
rotate ci keys
fathom-michael May 19, 2020
b96a815
Merge pull request #178 from medicode/michael/rotate-keys
fathom-michael May 19, 2020
66e205b
Update config.yml with new CircleCI Key
fathom-parth May 26, 2020
513f163
Merge pull request #181 from medicode/fathom-parth-patch-1
cbockman May 26, 2020
6d9b5a4
Update config.yml
peiranguo Jun 2, 2020
ded279d
Merge pull request #183 from medicode/peiranguo-patch-1
peiranguo Jun 2, 2020
62fbe2a
CircleCI key rotation 2020-06-09 (#187)
fathom-matthew Jun 9, 2020
550272d
CircleCI SA key rotation 2020-06-16 (#188)
fathom-matthew Jun 16, 2020
afb8d14
Update config.yml
keirn-fathom Jun 23, 2020
22113ec
Merge pull request #189 from medicode/keirn-fathom-patch-3
cbockman Jun 23, 2020
fca88d9
rotate CI key env var
fathom-michael Jul 2, 2020
ea14b50
Merge pull request #191 from medicode/michael/update-ci-key-var
cbockman Jul 2, 2020
d18501b
Rotate CircleCI SA keys
neogyzhu Jul 7, 2020
d690435
Merge pull request #193 from medicode/neogyzhu-patch-1
neogyzhu Jul 7, 2020
828f45d
CircleCI SA key rotation 2020-07-14 (#195)
fathom-matthew Jul 15, 2020
23cdd19
new key
peiranguo Jul 21, 2020
4d90f28
Merge pull request #196 from medicode/circleci-sa-key-rotation-2020-0…
cbockman Jul 21, 2020
a226c88
rotate keys
fathom-aditya Jul 28, 2020
eebdca8
Merge pull request #198 from medicode/aditya/buildcop
cbockman Jul 29, 2020
3c0c8fd
Cache problem in registry.
fathom-jakub Aug 4, 2020
File filter

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
reshape targets in eval
  • Loading branch information
rllin-fathom committed Sep 27, 2019
commit 6b88f99398afa84d68c3325bba51a6b7b7690aca
13 changes: 8 additions & 5 deletions tensor2tensor/utils/t2t_model.py
Original file line number Diff line number Diff line change
Expand Up @@ -1503,11 +1503,14 @@ def estimator_spec_eval(self, features, logits, labels, loss, losses_dict):
# like actions or rewards.
tf.logging.warning("No key %s in logits for evaluation." % k)
else:
# FATHOM
# NOTE: right now the tf.logging.warning will not trigger.
# ...consider if we should add
eval_metrics[metric_name] = metric_fn(logits, features,
features["targets"])
if hasattr(problem, 'max_docs_per_pack'):
features['targets'] = tf.reshape(features['targets'], [problem.max_docs_per_pack, -1])
else:
# FATHOM
# NOTE: right now the tf.logging.warning will not trigger.
# ...consider if we should add
eval_metrics[metric_name] = metric_fn(logits, features,
features["targets"])
if isinstance(logits, dict):
predictions = logits
else:
Expand Down