To run the script, at least following required packages should be satisfied:

• Python 3.5.2
• Tensorflow 1.6.0
• NumPy 1.14.1

You can use Anaconda to install these required packages. For tensorflow, use the following command to make a quick installation under windows:

pip install tensorflow 

In this project, there are implementations for various kinds of autoencoders. The base python class is library/Autoencoder.py, you can set the value of "ae_para" in the construction function of Autoencoder to appoint corresponding autoencoder.

• ae_para[0]: The corruption level for the input of autoencoder. If ae_para[0]>0, it's a denoising autoencoder;
• aw_para[1]: The coeff for sparse regularization. If ae_para[1]>0, it's a sparse autoencoder.

Follow the code sample below to construct a autoencoder:

corruption_level = 0 sparse_reg = 0 # n_inputs = 784 n_hidden = 400 n_outputs = 10 lr = 0.001 # define the autoencoder ae = Autoencoder(n_layers=[n_inputs, n_hidden], transfer_function = tf.nn.relu, optimizer = tf.train.AdamOptimizer(learning_rate = lr), ae_para = [corruption_level, sparse_reg]) 

To visualize the extracted features and images, check the code in visualize_ae.py.reconstructed

• Extracted features on MNIST:

• Reconstructed noisy images after input->encoder->decoder pipeline:

Follow the code sample below to construct a denoising autoencoder:

corruption_level = 0.3 sparse_reg = 0 # n_inputs = 784 n_hidden = 400 n_outputs = 10 lr = 0.001 # define the autoencoder ae = Autoencoder(n_layers=[n_inputs, n_hidden], transfer_function = tf.nn.relu, optimizer = tf.train.AdamOptimizer(learning_rate = lr), ae_para = [corruption_level, sparse_reg]) 

Test results:

• Extracted features on MNIST:

• Reconstructed noisy images after input->encoder->decoder pipeline:

Follow the code sample below to construct a sparse autoencoder:

corruption_level = 0 sparse_reg = 2 # n_inputs = 784 n_hidden = 400 n_outputs = 10 lr = 0.001 # define the autoencoder ae = Autoencoder(n_layers=[n_inputs, n_hidden], transfer_function = tf.nn.relu, optimizer = tf.train.AdamOptimizer(learning_rate = lr), ae_para = [corruption_level, sparse_reg]) 

For stacked autoencoder, there are more than one autoencoder in this network, in the script of "SAE_Softmax_MNIST.py", I defined two autoencoders:

corruption_level = 0.3 sparse_reg = 0 # n_inputs = 784 n_hidden = 400 n_hidden2 = 100 n_outputs = 10 lr = 0.001 # define the autoencoder ae = Autoencoder(n_layers=[n_inputs, n_hidden], transfer_function = tf.nn.relu, optimizer = tf.train.AdamOptimizer(learning_rate = lr), ae_para = [corruption_level, sparse_reg]) ae_2nd = Autoencoder(n_layers=[n_hidden, n_hidden2], transfer_function = tf.nn.relu, optimizer = tf.train.AdamOptimizer(learning_rate = lr), ae_para=[corruption_level, sparse_reg]) 

For the training of SAE on the task of MNIST classification, there are four sequential parts:

1. Training of the first autoencoder;
2. Training of the second autoencoder, based on the output of first ae;
3. Training on the output layer, normally softmax layer, based on the sequential output of first and second ae;
4. Fine-tune on the whole network.

Detailed code can be found in the script "SAE_Softmax_MNIST.py"

Class "autoencoder" are based on the tensorflow official models: https://github.com/tensorflow/models/tree/master/research/autoencoder/autoencoder_models

For the theory on autoencoder, sparse autoencoder, please refer to: http://ufldl.stanford.edu/tutorial/unsupervised/Autoencoders/

漫谈autoencoder：降噪自编码器/稀疏自编码器/栈式自编码器（含tensorflow实现）