backpropagation for (some) user-defined activation functions #58
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Note: requires the user to specify a "differential expression" for the activation function, by which I mean its derivative in terms of its function value. Thus limited to strictly increasing, differentiable functions.
pjvm742 changed the title Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
This simple change would half address issue #31 . It generalises the training procedure for the sigmoid to other invertible functions.
It doesn't work for most non-invertible activation functions.[a] Those could be addressed by a different training procedure which would require the weighted sum input to a node instead of just the resulting value, and would use the derivative proper instead of a "differential expression" in terms of the function value. I might implement this at some point, but it would be more of a hassle.
[a] one exception is ReLU, which is non-invertible as it has the same value for all negative inputs; it just so happens that the derivative also has the same value for all of those inputs.
I am unhappy with the naming - "differential expression" was the best I could come up with, hopefully someone knows a better term for this.
If there are any adjustments I need to make, please let me know.