Skip to content

Make fix_data_dir.sh remove utterances which have bad duration. #3275

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Apr 30, 2019
Merged

Make fix_data_dir.sh remove utterances which have bad duration. #3275

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion egs/wsj/s5/utils/fix_data_dir.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -155,12 +155,22 @@ function filter_utts {
maybe_reco2dur=
[ ! -f $data/segments ] && maybe_wav=wav.scp # wav indexed by utts only if segments does not exist.
[ -s $data/reco2dur ] && [ ! -f $data/segments ] && maybe_reco2dur=reco2dur # reco2dur indexed by utts
for x in feats.scp text segments utt2lang $maybe_wav; do

maybe_utt2dur=
if [ -f $data/utt2dur ]; then
cat $data/utt2dur | \
awk '{ if (NF == 2 && $2 > 0) { print }}' > $data/utt2dur.ok || exit 1
maybe_utt2dur=utt2dur.ok
fi

for x in feats.scp text segments utt2lang $maybe_wav $maybe_utt2dur; do
if [ -f $data/$x ]; then
utils/filter_scp.pl $data/$x $tmpdir/utts > $tmpdir/utts.tmp
mv $tmpdir/utts.tmp $tmpdir/utts
fi
done
rm $data/utt2dur.ok 2>/dev/null || true

[ ! -s $tmpdir/utts ] && echo "fix_data_dir.sh: no utterances remained: not proceeding further." && \
rm $tmpdir/utts && exit 1;

Expand Down