DEEP LEARNING
UNIT - III
By,
DR. HIMANI DESHPANDE (TSEC, MUMBAI) Dr. Himani Deshpande 1
UNIT – III AUTOENCODERS
3.1
Introduction, Linear Autoencoder, Undercomplete Autoencoder, Overcomplete
Autoencoders, Regularization in Autoencoders
3.2
Denoising Autoencoders, Sparse Autoencoders, Contractive Autoencoders
3.3
Application of Autoencoders: Image Compression
2
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
AUTOENCODERS
¡ An autoencoder is a type of artificial neural network used to learn efficient
codings of unlabeled data (unsupervised learning).
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
AUTOENCODERS
¡ Autoencoders are designed to reproduce their input, especially for images.
¡ Key point is to reproduce the input from a learned encoding.
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
AUTOENCODERS. ARCHITECTURE
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
AUTOENCODER
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
HIGHLIGHT NOTES
¡ The way we try to mark and learn
important things for exams instead of
learning the whole book chapter.
Autoencoder focuses on reproducing
significant information with some loss
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
AUTO ENCODERS
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
AUTOENCODERS
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
AUTOENCODERS
10
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
Why can't we just copy ?
AUTOENCODERS
That way latent layers
will not learn anything.
11
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
AUTOENCODERS
12
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
PCA AND AUTOENCODERS
13
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
PCA AND AUTOENCODERS
14
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
PCA AND AUTOENCODERS
15
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
APPLICATION
SELF DRIVING CARS
16
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
SELF DRIVING CARS
17
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
Properties of Autoencoders
DATA SPECIFIC
UNSUPERVISED
LOSSY
PROPERTIES OF AUTOENCODERS
• Autoencoders are data-specific, which means that they will only be able to compress
data similar to what they have been trained on.
• Example, an autoencoder trained on pictures of faces would do a rather poor job of
compressing pictures of trees, because the features it would learn would be face-specific.
• Autoencoders are lossy, which means that the decompressed outputs will be
degraded compared to the original inputs.
• Autoencoders are learned automatically from data examples, which is a useful
property: it means that it is easy to train specialized instances of the algorithm that will
perform well on a specific type of input. It doesn’t require any new engineering, just
appropriate training data. 20
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
PARTS OF AUTO ENCODERS
¡ Encoder : This part of the network encodes or compresses the input data into a
latent-space representation. The compressed data typically looks garbled, nothing
like the original data.
¡ Decoder : This part of network decodes or reconstructs the encoded data(latent
space representation) back to original dimension. The decoded data is a lossy
reconstruction of the original data.
21
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
AUTO-ENCODER
Usually <784
Compact
NN code representation of
Encoder the input object
28 X 28 = 784
Learn together
code NN
Decoder
Can reconstruct
the original object
22
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
AUTOENCODERS
Minimize 𝑥 − 𝑥# "
As close as possible
encode decode
𝑥 𝑐 𝑥#
𝑊 𝑊!
hidden layer
Input layer (linear) output layer
Bottleneck later
23
DR. HIMANI DESHPANDE (TSEC, MUMBAI) Output of the hidden layer is the code
TRAINING AUTOENCODERS
CODE SIZE
NUMBER OF LAYERS
NUMBER OF NODES
PER LAYER LOSS FUNCTION
TRAINING AUTOENCODERS
If we are working with image data, the most popular loss
functions for reconstruction are MSE Loss and L1 Loss.
In case the inputs and outputs are within the range [0,1], as in
MNIST, we can also make use of Binary Cross Entropy as the
reconstruction loss.
TRAINING AUTOENCODERS
You need to set 4 hyperparameters before training an autoencoder:
1.Code size: The code size or the size of the bottleneck is the most important hyperparameter used to tune the
autoencoder. The bottleneck size decides how much the data has to be compressed. This can also act as a
regularisation term.
2.Number of layers: Like all neural networks, an important hyperparameter to tune autoencoders is the depth
of the encoder and the decoder. While a higher depth increases model complexity, a lower depth is faster to
process.
3.Number of nodes per layer: The number of nodes per layer defines the weights we use per layer. Typically,
the number of nodes decreases with each subsequent layer in the autoencoder as the input to each of these
layers becomes smaller across the layers.
4.Reconstruction Loss: The loss function we use to train the autoencoder is highly dependent on the type of
input and output we want the autoencoder to adapt to. If we are working with image data, the most popular
loss functions for reconstruction are MSE Loss and L1 Loss. In case the inputs and outputs are within the range
[0,1], as in MNIST, we can also make use of Binary Cross Entropy as the reconstruction loss.
AUTO ENCODERS ARCHITECTURE
27
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
AUTOENCODER
¡ Autoencoders are identical to multilayer
perceptron neural networks because,
like multilayer perceptrons ,
autoencoders have an input layer, some
hidden layers, and an output layer.
¡ The key difference between a multilayer
perceptron network and an
autoencoder is that the output layer of
an autoencoder has the same number
of neurons as that of the input layer.
28
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
AUTOENCODER
h = g(W x i + b) The model is trained to minimize a certain loss function 29
xˆ i = f (W ∗ h + c) which will ensure that xˆi is close to x i
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
AUTO ENCODERS ARCHITECTURE
30
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
AUTO ENCODERS ARCHITECTURE
31
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
ENCODER
32
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
33
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
TYPES OF AE
9. Linear Autoencoder
10. Overcomplete Autoencoder 36
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
LINEAR AUTOENCODERS
¡ A linear autoencoder is a type of autoencoder that uses only linear transformations,
such as matrix multiplication and addition, to compress and reconstruct the data.
¡ A linear autoencoder consists of two parts: an encoder and a decoder. The encoder
takes the input data and maps it to a lower-dimensional space. The decoder then
takes the compressed representation and reconstructs the original data.
¡ The goal of training a linear autoencoder is to minimize the reconstruction error
between the input and output:
L(x, x') = ||x - x'||^2
37
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
LINEAR AUTOENCODERS
¡ A linear autoencoder is a type of autoencoder that uses only linear transformations.
In other words, the encoder and decoder are composed of only linear layers. The
advantage of using a linear autoencoder is that it is computationally efficient and can
be trained on large datasets.
38
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
LINEAR AUTOENCODERS
39
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
UNDERCOMPLETE AUTOENCODER
40
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
UNDERCOMPLETE AUTOENCODER
Ø Let us consider the case where
dim(h) < dim(x i )
Ø If we are still able to reconstruct xˆi perfectly
from h, then what does it say about h?
Ø h is a loss-free encoding of x i . It cap- tures all
the important characteristics of x i
An autoencoder where dim(h) < dim(x i ) is called an
under-complete autoencoder
41
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
UNDERCOMPLETE AUTOENCODERS
¡ An undercomplete autoencoder is one of the simplest types of autoencoders.
¡ The way it works is very straightforward—
Undercomplete autoencoder takes in an image and tries to predict the same image
as output, thus reconstructing the image from the compressed bottleneck region.
¡ Undercomplete autoencoders are truly unsupervised as they do not take any form of
label, the target being the same as the input.
42
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
UNDERCOMPLETE AE
43
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
OVERCOMPLETE AUTOENCODER
44
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
APPLICATION
¡ Applications of undercomplete autoencoders include compression, recommendation
systems as well as outlier detection.
45
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
OVERCOMPLETE AUTOENCODER
46
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
OVERCOMPLETE AUTOENCODER
Ø In such a case the autoencoder could learn a
trivial encoding by simply copying x i into h
and then copying h into xˆi
Ø Such an identity encoding is useless in
practice as it does not really tell us anything
about the important characteristics of the
data
An autoencoder where dim(h) ≥ dim(x i) 47
is called an over complete autoencoder
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
APPLICATION
¡ Very rare applications of overcomplete autoencoder, hypothetical scenarios.
¡ BMI à Height & Weight calculation.
¡ Sometimes inspite of knowing BMI, we might need our network to gain knowledge about height or weight.
Sparse Autoencoder based on regularization
48
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
UNDERCOMPLETE AND OVERCOMPLETE AUTOENCODERS
ARCHITECTURE
The only difference between the two is in the encoding
output's size 49
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
UNDERCOMPLETE AND OVERCOMPLETE AUTOENCODERS
¡ We can make out latent space representation learn useful features by giving it smaller
dimensions then input data. In this case autoencoder is undercomplete. By training an
undercomplete representation, we force the autoencoder to learn the most salient features
of the training data. If we give autoencoder much capacity(like if we have almost same
dimensions for input data and latent space), then it will just learn copying task without
extracting useful features or information from data.
¡ If dimensions of latent space is equal to or greater then to input data, in such case
autoencoder is overcomplete. In such case even linear encoder and linear decoder can learn
to copy the input to the output without learning anything useful about the data distribution.
50
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
STACK AUTOENCODER
A stacked autoencoder is a
neural network consist several
layers of sparse autoencoders
where output of each hidden
layer is connected to the input
of the successive hidden layer.
51
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
Regularization in
Autoencoders
It is fine to
loose some
data in this
case
52
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
REGULARIZATION
¡ A reliable autoencoder must make a tradeoff between two important parts:
• Sensitive enough to inputs so that it can accurately reconstruct input data
• Able to generalize well even when evaluated on unseen data
¡ As a result, our loss function of autoencoder is composed of two different parts.
¡ The first part is the loss function (e.g. mean squared error loss) calculating the difference
between input data and output data.
¡ The second term would act as regularization term which prevents autoencoder from
overfitting.
53
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
REGULARIZATION
¡ Regularization helps with the effects of out-of-control parameters by using different methods to
minimize parameter size over time.
¡ Regularization coefficients L1 and L2 help fight overfitting by making certain weights smaller. Smaller-
valued weights lead to simpler hypotheses, which are the most generalizable.
¡ Unregularized weights with several higher-order polynomials in the feature sets tend to overfit the
training set.
54
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
R EG U L A R I ZAT I O N
ü While poor generalization could happen
even in undercomplete autoencoders it is an
even more serious problem for overcomplete
auto encoders
ü Here, the model can simply learn to copy
x i to h and then h to xˆi
ü To avoid poor generalization, we need to
introduce regularization
55
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
R EG U L A R I ZAT I O N
¡ The simplest solution is to add a
L 2 -regularization term to the objective
function
‘m’ is the number of rows , n is number of
columns in the X vector of image
56
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
R EG U L A R I ZAT I O N
¡ The simplest solution is to add a
L 2 -regularization term to the objective
function
57
Θ= combination of all the weights and bias
Θ = [ w1,w2,w3,….. ]
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
REGULARIZATION
¡ The regularized autoencoders use a loss function that helps the model to have other
properties besides copying input to the output.
¡ We can generally find two types of regularized autoencoder:
¡ the denoising autoencoder and
¡ the sparse autoencoder.
58
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
DENOISING AUTOENCODERS
¡ Autoencoders are Neural Networks which are commonly used for feature selection and extraction.
However, when there are more nodes in the hidden layer than there are inputs, the Network is risking
to learn the so-called “Identity Function”, also called “Null Function”, meaning that the output equals
the input, marking the Autoencoder useless.
¡ Denoising Autoencoders solve this problem by corrupting the data on purpose by randomly turning
some of the input values to zero. In general, the percentage of input nodes which are being set to zero
is about 50%. Other sources suggest a lower count, such as 30%. It depends on the amount of data
and input nodes you have.
¡ When calculating the Loss function, it is important to compare the output values with the original
input, not with the corrupted input. That way, the risk of learning the identity function instead of
extracting features is eliminated.
59
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
DENOISING
AUTOENCODERS
¡ Denoising autoencoders are a robust variant of the standard autoencoders.
¡ They have the same structure as a standard autoencoders but are trained using
samples in which some amount of noise is added.
¡ Thus, we map these noisy samples to their clean version.
¡ This ensures that the network doesn’t learn an identity mapping which will be
pointless.
¡ So, to summarise, denoising autoencoders are used where you want to learn a more
robust latent representation for particular set of input data. 60
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
DENOISING AUTOENCODER
61
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
DENOISING AUTOENCODER
62
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
SPARSE AUTOENCODER
¡ A sparse autoencoder is simply an autoencoder whose training criterion involves a sparsity
penalty. In most cases, we would construct our loss function by penalizing activations of
hidden layers so that only a few nodes are encouraged to activate when a single sample is
fed into the network.
¡ There are actually two different ways to construct our sparsity penalty:
¡ L1 regularization and
¡ KL-divergence.
69
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
SPARSE AUTOENCODER
70
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
SPARSE AUTOENCODER
71
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
SPARSE AUTOENCODER
¡ In sparse autoencoders with a sparsity enforcer that directs a single layer network to learn
code dictionary which minimizes the error in reproducing the input while constraining
number of code words for reconstruction.
¡ The sparse autoencoder consists of a single hidden layer, which is connected to the input
vector by a weight matrix forming the encoding step. The hidden layer outputs to a
reconstruction vector, using a tied weight matrix to form the decoder.
73
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
CONTRACTIVE AE
74
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
CONTRACTIVE AUTOENCODER
¡ A Contractive Autoencoder is an autoencoder that adds a penalty term to the classical reconstruction cost
function.
¡ This penalty term corresponds to the Frobenius norm of the Jacobian matrix of the encoder activations with
respect to the input.
The Frobenius Norm of a matrix is defined as the
square root of the sum of the squares of the
elements of the matrix. Approach: Find the sum
of squares of the elements of the matrix and
then print the square root of the calculated
value.
75
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
CONTRACTIVE AUTOENCODER
¡ Contractive Autoencoder was proposed by the researchers at the University of Toronto in 2011 in the
paper Contractive auto-encoders: Explicit invariance during feature extraction. The idea behind that is
to make the autoencoders robust of small changes in the training dataset.
¡ To deal with the above challenge that is posed in basic autoencoders, the authors proposed to add
another penalty term to the loss function of autoencoders.
¡ The Loss function:
Contractive autoencoder adds an extra term in the loss function of autoencoder, it is given as:
i.e the above penalty term is the Frobenius Norm of the encoder, the frobenius norm is just a
generalization of Euclidean norm. 76
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
DENOISING AND CONTRACTIVE AUTOENCODER
¡ There is a connection between the denoising autoencoder and the contractive autoencoder:
¡ the denoising reconstruction error is equivalent to a contractive penalty on the reconstruction function that
maps x to r - g(f(x)).
¡ In other words, denoising autoencoders make the reconstruction function resist small but finite sized
perturbations of the input, whereas contractive autoencoders make the feature extraction function resist
infinitesimal perturbations of the input.
77
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
Symmetric is not
necessary.
DEEP AUTO-ENCODER
¡ Of course, the auto-encoder can be deep
As close as possible
Output Layer
Input Layer
bottle
… …
Layer
Layer
Layer
Layer
Layer
Layer
𝑊# 𝑊" 𝑊"! 𝑊#!
𝑥 Initialize by RBM 𝑥#
Code 78
layer-by-layer
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
85
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
APPLICATIONS
WATERMARK
REMOVAL
01
DENOISING
05
02 DIMENTIONALITY
REDUCTION
IMAGE
COMPRESSION
IMAGE 04 03 FEATURE
COLORIZATION
VARIATION
USE OF AUTO ENCODERS
¡ Data denoising and Dimensionality reduction for data visualization are considered
as two main interesting practical applications of autoencoders. With appropriate
dimensionality and sparsity constraints, autoencoders can learn data projections that
are more interesting than PCA or other basic techniques.
¡ Autoencoders also can be used for Image Reconstruction, Basic Image colorization,
data compression, gray-scale images to colored images, generating higher resolution
images etc.
87
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
APPLICATIONS
Autoencoders present an efficient way to learn a representation of
your data that focuses on the signal, not the noise. You can use them
for a variety of tasks such as:
•Image Compression
•Dimensionality reduction
•Feature extraction
•Denoising of data/images
•Imputing missing data
IMAGE COMPRESSION
89
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
IMAGE COMPRESSION
¡ Autoencoders are a deep learning model for transforming data from a high-
dimensional space to a lower-dimensional space. They work by encoding the data,
whatever its size, to a 1-D vector. This vector can then be decoded to reconstruct
the original data (in this case, an image).
¡ An autoencoder consists of two parts: an encoder network and a decoder network.
The encoder network compresses the input data, while the decoder network
reconstructs the compressed data back into its original form. The compressed data,
also known as the bottleneck layer, is typically much smaller than the input data.
90
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
AUTOENCODERS: APPLICATIONS
¡ Denoising: input clean image + noise and train to reproduce the clean image.
91
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
AUTOENCODERS: APPLICATIONS
¡ Image colorization: input black and white and train to produce color images
92
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
AUTOENCODERS: APPLICATIONS
¡ Watermark removal
93
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
FEATURE VARIATION
94
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
DIMENSIONALITY REDUCTION
95
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
96
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
PROPERTIES OF AUTOENCODERS
¡ Data-specific: Autoencoders are only able to compress data similar to what they have been trained on.
¡ Lossy: The decompressed outputs will be degraded compared to the original inputs.
¡ Learned automatically from examples: It is easy to train specialized instances of the algorithm that will
perform well on a specific type of input.
97
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
https://www.edureka.co/blog/autoencoders-tutorial/
CAPACITY
¡ As with other NNs, overfitting is a problem when capacity is too large for the data.
¡ Autoencoders address this through some combination of:
¡ Bottleneck layer – fewer degrees of freedom than in possible outputs.
¡ Training to denoise.
¡ Sparsity through regularization.
¡ Contractive penalty.
98
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
BOTTLENECK LAYER (UNDERCOMPLETE)
¡ Suppose input images are nxn and the latent space is m < nxn.
¡ Then the latent space is not sufficient to reproduce all images.
¡ Needs to learn an encoding that captures the important features in training data, sufficient for approximate
reconstruction.
99
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
SIMPLE BOTTLENECK LAYER IN KERAS
¡ input_img = Input(shape=(784,))
¡ encoding_dim = 32
¡ encoded = Dense(encoding_dim, activation='relu')(input_img)
¡ decoded = Dense(784, activation='sigmoid')(encoded)
¡ autoencoder = Model(input_img, decoded)
¡ Maps 28x28 images into a 32 dimensional vector.
¡ Can also use more layers and/or convolutions.
100
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
https://blog.keras.io/building-autoencoders-in-keras.html
DENOISING AUTOENCODERS
¡ Basic autoencoder trains to minimize the loss between x and the reconstruction g(f(x)).
¡ Denoising autoencoders train to minimize the loss between x and g(f(x+w)), where w is random noise.
¡ Same possible architectures, different training data.
¡ Kaggle has a dataset on damaged documents.
101
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
https://blog.keras.io/building-autoencoders-in-keras.html
DENOISING AUTOENCODERS
¡ Denoising autoencoders can’t simply memorize the input output relationship.
¡ Intuitively, a denoising autoencoder learns a projection from a neighborhood of our
training data back onto the training data.
102
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
SPARSE AUTOENCODERS
¡ Construct a loss function to penalize activations within a layer.
¡ Usually regularize the weights of a network, not the activations.
¡ Individual nodes of a trained model that activate are data-dependent.
¡ Different inputs will result in activations of different nodes through the network.
¡ Selectively activate regions of the network depending on the input data.
103
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
https://www.jeremyjordan.me/autoencoders/
SPARSE AUTOENCODERS
¡ Construct a loss function to penalize activations the network.
¡ L1 Regularization: Penalize the absolute value of the vector of activations a in layer h for observation I
¡ KL divergence: Use cross-entropy between average activation and desired activation
104
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
https://www.jeremyjordan.me/autoencoders/
CONTRACTIVE AUTOENCODERS
¡ Arrange for similar inputs to have similar activations.
¡ I.e., the derivative of the hidden layer activations are small with respect to the input.
¡ Denoising autoencoders make the reconstruction function (encoder+decoder) resist
small perturbations of the input
¡ Contractive autoencoders make the feature extraction function (ie. encoder) resist
infinitesimal perturbations of the input.
105
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
https://www.jeremyjordan.me/autoencoders/
CONTRACTIVE AUTOENCODERS
¡ Contractive autoencoders make the feature extraction function (ie. encoder) resist infinitesimal perturbations of
the input.
106
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
https://ift6266h17.files.wordpress.com/2017/03/14_autoencoders.pdf
AUTOENCODERS
¡ Both the denoising and contractive autoencoder can perform well
¡ Advantage of denoising autoencoder : simpler to implement-requires adding one or two lines of code to regular
autoencoder-no need to compute Jacobian of hidden layer
¡ Advantage of contractive autoencoder : gradient is deterministic -can use second order optimizers (conjugate gradient,
LBFGS, etc.)-might be more stable than denoising autoencoder, which uses a sampled gradient
¡ To learn more on contractive autoencoders:
¡ Contractive Auto-Encoders: Explicit Invariance During Feature Extraction. Salah Rifai, Pascal Vincent, Xavier Muller, Xavier
Glorot et Yoshua Bengio, 2011.
107
DR. HIMANI DESHPANDE (TSEC, MUMBAI)
https://ift6266h17.files.wordpress.com/2017/03/14_autoencoders.pdf
