Skip to content

Fix the input_fn for multi-node training #386

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
changlan wants to merge 6 commits into
base: master
Choose a base branch
from
Open

Fix the input_fn for multi-node training #386

changlan wants to merge 6 commits into from

Conversation

@changlan
Copy link

@changlan changlan commented Jan 27, 2020

  • Do not create sharded TFRecords.
  • Ensure there is only one process per host that creates the TFRecord file.

Signed-off-by: Chang Lan changlan@google.com
@changlan
Copy link
Author

changlan commented Jan 28, 2020

Fix #383. The issue I had was that host 0 had shard 0-7 and host 1 had shard 8-15, but TFRecordDataset on every MPI process had to access all the shards regardless of d.shard(). I did not observe any performance issue after removing all the shards.
@changlan changlan requested a review from nvpstr January 31, 2020 00:20
@changlan changlan mentioned this pull request Feb 10, 2020
Closed
@nvpstr nvpstr requested review from swethmandava and removed request for nvpstr March 9, 2020 16:45
@nv-kkudrynski nv-kkudrynski requested review from meatybobby and removed request for swethmandava August 13, 2021 22:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

3 participants

@changlan @meatybobby @swethmandava