Attention modeling, with example scripts

[danpovey]

still far from compiling.

[danpovey]

@hhadian, there's some stuff you can help with here:

• Add attention.o to Makefile and make sure it compiles.
• Write test for GetAttentionDotProducts() [you can do this with reference to the
  comment that explains what it does: write a simple version in the test file, that
  you can compare with.]
• Write ApplyScalesToOutput() and ApplyScalesToInput(), and write testing code for them.
  The code will have a loop that goes up to context_dim, just like GetAttentionDotProducts(), and
  each time it loops it will call AddDiagVecMat. It will need to create a temporary transposed copy
  of the 'C' input.

Note: eventually, if this works, we may ask @kangshiyin to write CUDA versions of GetAttentionDotProducts(), ApplyScalesToOutput(), and ApplyScalesToInput(). But this probably won't take more than half the time, even with the naive implementation, so there's no need to do that just yet.

[hhadian]

Will do

[hhadian]

Is TimeHeightConvolutionComponent supposed to be here? Almost all the functions ralating TimeHeightConvolutionComponent are duplicated here.

[danpovey]

I am copying-and-modifying that file. I'll remove all that stuff. For now don't bother compiling that file, I'm working on it.

…

On Fri, Jun 30, 2017 at 4:03 PM, Hossein Hadian ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In src/nnet3/nnet-attention-component.cc <#1731 (comment)>: > + // 'keys' contains the keys; note, these are not extended with + // context information; that happens further in. + CuSubMatrix<BaseFloat> keys(in, 0, in.NumRows(), 0, key_dim_); + + // 'values' contains the values which we will interpolate. + // these don't contain the context information; that will be added + // later if output_context_ == true. + CuSubMatrix<BaseFloat> values(in, 0, in.NumRows(), key_dim_, value_dim_); + + + AttentionForward(key_scale_, keys, queries, values, c, out); +} + + + +void TimeHeightConvolutionComponent::Check() { Is TimeHeightConvolutionComponent supposed to be here? Almost all the functions ralating TimeHeightConvolutionComponent are duplicated here. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1731 (review)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADJVuxQTo75lwzns5X8feBthBKn5AW6iks5sJVR3gaJpZM4OK40U> .

[hhadian]

OK, sure, I was just wondering.

[hhadian]

Do you want me to write the test for GetAttentionDotProducts in a new file attention-test.cc or in an existing tester file? Also, the function itself seems not to be implemented yet (I looked in attention.cc), should I implement it?

I read the docs, but I'm not sure what query, key, and value are going to be in ASR tasks. The values should be the frames of speech, but what are the keys and queries?

[danpovey]

new file attention-test.cc. It's implemented but mis-named as AttentionCoreForward(), in the cc file, fix the name. query, key and value are all just different sub-ranges of the input to the layer, i.e. they're ranges of column indexes.

…

On Fri, Jun 30, 2017 at 5:47 PM, Hossein Hadian ***@***.***> wrote: Do you want me to write the test for GetAttentionDotProducts in a new file attention-test.cc or in an existing tester file? Also, the function itself seems not to be implemented yet (I looked in attention.cc), should I implement it? I read the docs, but I'm not sure what query, key, and value are going to be in ASR tasks. The values should be the frames of speech, but what are the keys and queries? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1731 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADJVuwoqpCeqBmWKm8Smyl9vUU6kFPTOks5sJWz4gaJpZM4OK40U> .

[hhadian]

This line has j on the right-hand side but not on the left-hand side. I'm not sure I get it right.

[danpovey]

add a \sum_j on the right hand side.

…

On Fri, Jun 30, 2017 at 8:23 PM, Hossein Hadian ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In src/nnet3/attention.h <#1731 (comment)>: > + CuMatrixBase<BaseFloat> *C); + + +/** + This function is related to GetAttentionDotProducts(); it + is used in scaling the values by the softmax scales, and + in backprop. + + We have put the A, B and C in an unusual order here in order + to make clearer the relationship with GetAttentionDotProducts(). + The matrices have the same relationship in terms of their + dimensions, as A, B and C do in GetAttentionDotProducts(). + + This function implements: + + A->Row(i) += alpha * C(i, j) * B.Row(i + j * row_shift). This line has j on the right-hand side but not on the left-hand side. I'm not sure I get it right. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1731 (review)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADJVu1IcinmZoX2aUvuYZpKkx5AU0Rj2ks5sJZGDgaJpZM4OK40U> .

[hhadian]

Since it is asserted (a few lines before) that values and c both have num_output_rows rows, this will set A, B, and C in a way that all have the same number of rows, so row_shift will become 0.

[danpovey]

'values' should have 'num_input_rows' rows, the assert must have been a mistake.

…

On Fri, Jun 30, 2017 at 11:26 PM, Hossein Hadian ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In src/nnet3/attention.cc <#1731 (comment)>: > + queries_key_part, + keys, c); + // think of 'queries_context_part' as a position-dependent bias term. + c->AddMat(1.0, queries_context_part); + // compute the soft-max function. Up till this point, 'c' + // actually contained what in attention.h we called 'b', which is + // the input to the softmax. + c->SoftMaxPerRow(*c); + + + // the part of the output that is weighted + // combinations of the input values. + CuSubMatrix<BaseFloat> output_values_part( + *output, 0, num_output_rows, 0, value_dim); + + ApplyScalesToOutput(1.0, values, *c, &output_values_part); Since it is asserted (a few lines before) that values and c both have num_output_rows rows, this will set A, B, and C in a way that all have the same number of rows, so row_shift will become 0. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1731 (review)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADJVuyCMB9aaXxZPvPYxJfoae7RRSjMEks5sJbxmgaJpZM4OK40U> .

[danpovey]

@hhadian, I think the component-level code is now working and tested.

Can you please work on the script-level changes required to test this?
I think the easiest way to do this will be to copy-and-modify BasicLayer, supporting layers of the
form affine + attention + [some kind of nonlinearity]. Do this in a new file attention.py Let's not bother with the ResNet-like thing they were doing, at this point; I've found previously in speech tasks that that stuff was not helpful. Note: our "NormalizeLayer" is basically the same as the "layer normalization" from Hinton that they refer to in the paper.

E.g. we want someone to be able to write in a config line:

attention-renorm-layer num-heads=10 value-dim=50 key-dim=50 time-stride=3 num-left-inputs=5 num-right-inputs=2.
or
attention-relu-renorm-layer num-heads=10 value-dim=50 key-dim=50 time-stride=3 num-left-inputs=5 num-right-inputs=2.

You can intersperse these with regular relu-batchnorm-layers for initial experiments.

You can have num-left-inputs-required and num-right-inputs-required and key-dim all present but defaulting to -1, and output-context present and defaulting to true; time-stride can default to 1 and num-heads to 1, but require the user to specify value-dim, key-dim, num-left-inputs and num-right-inputs.

[hhadian]

Will do

[hhadian]

What values do you suggest for the first experiment? With num-heads=10 value-dim=50 key-dim=50 time-stride=3 num-left-inputs=5 num-right-inputs=2, the input dim of the attention block is 1580 and the output dim is 580.

[danpovey]

That's similar to TDNNs with relu-dim=512, so I think it's a reasonable setting.

…

On Mon, Jul 3, 2017 at 7:30 PM, Hossein Hadian ***@***.***> wrote: What values do you suggest for the first experiment? With num-heads=10 value-dim=50 key-dim=50 time-stride=3 num-left-inputs=5 num-right-inputs=2, the input dim of the attention block is 1580 and the output dim is 580. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1731 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADJVu-4j3R2kPZ6Te2SEwqhOt2O3sP2zks5sKXmpgaJpZM4OK40U> .

[danpovey]

@hhadian, I notice that the stats are not being printed in the progress logs.
It is due to an oversight on my part, I should have implemented the functions Add() and Scale() in the component, which would add and scale the stats. Also ZeroStats(). Can you fix it? You can look at class NonlinearComponent (see nnet-component-itf.{h,cc}) for inspiration.

[hhadian]

Sure, will do.

[kkm000]

Optionally, use std::set, which is sorted. Might be marginally more efficient.

You may also use some stl magic to reduce the function to just three lines, arguably more readable:

std::set<int32> t_set; std::remove_copy(indexes.begin(), indexes.end(), std::inserter(t_set, t_set.begin()), kNoTime); t_values->assign(t_set.begin(), t_set.end());

(include <algorithm> and <iterator>).

[danpovey]

In the normal case, there could be many (e.g. 128) copies of each 't' value, so in that case I think
the way we have it is more efficient. (Also Index is a struct, not an integer).

[kkm000]

I see, missed the iter->t part.

[hhadian]

The results of changing num_heads while keeping the output/input dimension constant.
[The input dim of attention layer is num_heads * (3 * dim + C) assuming key_dim = value_dim = dim and C is context-size]
[The output dim is num_heads * (dim + C) ]

In the following results, input_dim ~= 1580

# System tdnn_7k head15_dim34 head10_dim50 head8_dim63 # WER on train_dev(tg) 13.93 14.33 14.26 14.10 # WER on train_dev(fg) 12.85 13.27 12.93 12.99 # WER on eval2000(tg) 16.7 16.9 16.6 16.6 # WER on eval2000(fg) 15.0 15.2 14.8 15.0 # Final train prob -0.085 -0.079 -0.079 -0.080 # Final valid prob -0.106 -0.103 -0.102 -0.101 # Final train prob (xent) -1.260 -1.026 -1.024 -1.037 # Final valid prob (xent) -1.3193 -1.1072 -1.1048 -1.1147 

In the following results, input_dim ~= 2330 (i.e. ~50% bigger layers):

# System tdnn_7k head20_dim37 head15_dim50 head10_dim75 # WER on train_dev(tg) 13.93 14.07 13.96 14.01 # WER on train_dev(fg) 12.85 12.98 12.90 12.81 # WER on eval2000(tg) 16.7 16.9 16.4 16.4 # WER on eval2000(fg) 15.0 15.3 14.8 14.9 # Final train prob -0.085 -0.076 -0.078 -0.077 # Final valid prob -0.106 -0.100 -0.101 -0.101 # Final train prob (xent) -1.260 -0.999 -0.995 -1.003 # Final valid prob (xent) -1.3193 -1.0964 -1.0946 -1.0926 

In all these, there are 2 attention layers, one near the beginning and one near the end and context is (5, 2).

[danpovey]

Hm. It does look like having the key/value dimension less than about 50 is harmful. Try setups where the value dimension is larger than the key dimension (e.g. twice larger), with slightly fewer heads. If you could do some of these experiments in a context where you replace more of the TDNN layers with attention layers, the results might be more different, even if we ultimately decide to use only a couple of attention layers.

…

On Sat, Jul 15, 2017 at 1:08 AM, Hossein Hadian ***@***.***> wrote: The results of changing num_heads while keeping the output/input dimension constant. [The input dim of attention layer is num_heads * (3 * dim + C) assuming key_dim = value_dim = dim and C is context-size] [The output dim is num_heads * (dim + C) ] In the following results, input_dim ~= 1580 # System tdnn_7k head15_dim34 head10_dim50 head8_dim63 # WER on train_dev(tg) 13.93 14.33 14.26 14.10 # WER on train_dev(fg) 12.85 13.27 12.93 12.99 # WER on eval2000(tg) 16.7 16.9 16.6 16.6 # WER on eval2000(fg) 15.0 15.2 14.8 15.0 # Final train prob -0.085 -0.079 -0.079 -0.080 # Final valid prob -0.106 -0.103 -0.102 -0.101 # Final train prob (xent) -1.260 -1.026 -1.024 -1.037 # Final valid prob (xent) -1.3193 -1.1072 -1.1048 -1.1147 In the following results, input_dim ~= 2330 (i.e. ~50% bigger layers): # System tdnn_7k head20_dim37 head15_dim50 head10_dim75 # WER on train_dev(tg) 13.93 14.07 13.96 14.01 # WER on train_dev(fg) 12.85 12.98 12.90 12.81 # WER on eval2000(tg) 16.7 16.9 16.4 16.4 # WER on eval2000(fg) 15.0 15.3 14.8 14.9 # Final train prob -0.085 -0.076 -0.078 -0.077 # Final valid prob -0.106 -0.100 -0.101 -0.101 # Final train prob (xent) -1.260 -0.999 -0.995 -1.003 # Final valid prob (xent) -1.3193 -1.0964 -1.0946 -1.0926 In all these, there are 2 attention layers, one near the beginning and one near the end and context is (5, 2). — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1731 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADJVuwqfaonr_M-hFzRznCi1g2g3iHx_ks5sOElogaJpZM4OK40U> .

[hhadian]

Will do

[hhadian]

Results regarding the position of attention layer in the network:

# System tdnn_7k L2 L5 L6 L7 # WER on train_dev(tg) 13.93 14.11 14.00 13.80 13.64 # WER on train_dev(fg) 12.85 12.94 12.85 12.74 12.55 # WER on eval2000(tg) 16.7 16.8 16.6 16.4 16.3 # WER on eval2000(fg) 15.0 15.2 15.0 15.0 14.8 # Final train prob -0.085 -0.085 -0.080 -0.079 -0.077 # Final valid prob -0.106 -0.104 -0.101 -0.101 -0.099 # Final train prob (xent) -1.260 -1.150 -1.034 -1.030 -1.009 # Final valid prob (xent) -1.3193 -1.2133 -1.1319 -1.1034 -1.0980 

Li means only layer i is attention and the rest are TDNN as baseline.
With the current config (i.e. context 5,2, key/value dim 50 and num-heads 10 ) attention is not working in the initial layers (I used time-stride=1 for L2 and time-stride=3 for the rest). I am trying a larger context for L2 to see if it helps.
I guess I should also try it on pre-final-chain (i.e. layer 8).

[hhadian]

Since attention is good at layer 7, I tried bigger value dimensions with that:

# System tdnn_7k L7_key50_val50 L7_key50_val100 L7_key40-val80 # WER on train_dev(tg) 13.93 13.64 13.64 13.76 # WER on train_dev(fg) 12.85 12.55 12.68 12.62 # WER on eval2000(tg) 16.7 16.3 16.3 16.2 # WER on eval2000(fg) 15.0 14.8 14.7 14.6 # Final train prob -0.085 -0.077 -0.074 -0.076 # Final valid prob -0.106 -0.099 -0.095 -0.098 # Final train prob (xent) -1.260 -1.009 -0.984 -0.997 # Final valid prob (xent) -1.3193 -1.0980 -1.0727 -1.0887 

So the best result is currently L7_key40-val80 with 0.5% and 0.4% absolute improvement on eval2000 tg and fg.

[danpovey]

Cool! Don't ignore the train_dev, those numbers are just as important as eval2000. Overall it's not clear to me that you are getting an improvement from a larger value-dim. Try a larger amount of left and right context for the attention layer, or two attention layers right at the end.

…

On Tue, Jul 18, 2017 at 3:54 PM, Hossein Hadian ***@***.***> wrote: Since attention is good at layer 7, I tried bigger value dimensions with that: # System tdnn_7k L7_key50_val50 L7_key50_val100 L7_key40-val80 # WER on train_dev(tg) 13.93 13.64 13.64 13.76 # WER on train_dev(fg) 12.85 12.55 12.68 12.62 # WER on eval2000(tg) 16.7 16.3 16.3 16.2 # WER on eval2000(fg) 15.0 14.8 14.7 14.6 # Final train prob -0.085 -0.077 -0.074 -0.076 # Final valid prob -0.106 -0.099 -0.095 -0.098 # Final train prob (xent) -1.260 -1.009 -0.984 -0.997 # Final valid prob (xent) -1.3193 -1.0980 -1.0727 -1.0887 So the best result is currently L7_key40-val80 with 0.5% and 0.4% absolute improvement on eval2000 tg and fg. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1731 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADJVu3fCGO3TKQZzLXy9A7j29xWD88uhks5sPQ2AgaJpZM4OK40U> .

[hhadian]

Will do.
Re train_dev, is it really just as important as eval2000? because my impression was that train_sev has a lot of speaker overlap and should be considered less important when tuning.

[danpovey]

I think even the test sets have speaker overlap on Switchboard. The only time I'd be concerned about speaker overlap is in things that are specifically about speaker adaptation. These changes are orthogonal to that and I'm more concerned about test-set noise due to limited size.

…

On Tue, Jul 18, 2017 at 5:52 PM, Hossein Hadian ***@***.***> wrote: Will do. Re train_dev, is it really just as important as eval2000? because my impression was that train_sev has a lot of speaker overlap and should be considered less important when tuning. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1731 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADJVu88eq2uPTv1sa19mUcpeqLi8DBbYks5sPSj-gaJpZM4OK40U> .

[danpovey]

@hhadian, sorry, your authorship seems to have been lost by git due to the squash (I don't like to merge, except between versions of Kaldi). Next time you merge stuff, it will be to master anyway.