Replace all occurrences of auto-encoder with autoencoder

Author: krasserm

README.md

@@ -7,13 +7,13 @@ the notebooks via [nbviewer](https://nbviewer.jupyter.org/) to ensure a proper r
 PyMC3 and PyMC4 implementations are now available for some notebooks (more planned).
 
 - [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/krasserm/bayesian-machine-learning/blob/master/latent_variable_models_part_1.ipynb)
- [Latent variable models - part 1: Gaussian mixture models and the EM algorithm](https://nbviewer.jupyter.org/github/krasserm/bayesian-machine-learning/blob/master/latent_variable_models_part_1.ipynb).
+ [Latent variable models, part 1: Gaussian mixture models and the EM algorithm](https://nbviewer.jupyter.org/github/krasserm/bayesian-machine-learning/blob/master/latent_variable_models_part_1.ipynb).
  Introduction to the expectation maximization (EM) algorithm and its application to Gaussian mixture models. Example
  implementation with plain NumPy/SciPy and scikit-learn for comparison (see also 
  [PyMC3 implementation](https://nbviewer.jupyter.org/github/krasserm/bayesian-machine-learning/blob/master/latent_variable_models_part_1_pymc3.ipynb)).
 
 - [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/krasserm/bayesian-machine-learning/blob/master/latent_variable_models_part_2.ipynb)
- [Latent variable models - part 2: Stochastic variational inference and variational autoencoders](https://nbviewer.jupyter.org/github/krasserm/bayesian-machine-learning/blob/master/latent_variable_models_part_2.ipynb). 
+ [Latent variable models, part 2: Stochastic variational inference and variational autoencoders](https://nbviewer.jupyter.org/github/krasserm/bayesian-machine-learning/blob/master/latent_variable_models_part_2.ipynb). 
  Introduction to stochastic variational inference with variational autoencoder as application example. Implementation 
  with Tensorflow 2.x.
 
@@ -36,10 +36,10 @@ PyMC3 and PyMC4 implementations are now available for some notebooks (more plann
  Introduction to Bayesian optimization. Example implementations with plain NumPy/SciPy as well as with libraries 
  scikit-optimize and GPyOpt. Hyperparameter tuning as application example. 
 
-- [Deep feature consistent variational auto-encoder](https://nbviewer.jupyter.org/github/krasserm/bayesian-machine-learning/blob/master/variational_autoencoder_dfc.ipynb). 
- Describes how a perceptual loss can improve the quality of images generated by a variational auto-encoder. Example 
+- [Deep feature consistent variational autoencoder](https://nbviewer.jupyter.org/github/krasserm/bayesian-machine-learning/blob/master/variational_autoencoder_dfc.ipynb). 
+ Describes how a perceptual loss can improve the quality of images generated by a variational autoencoder. Example 
  implementation with Keras. 
 
 - [Conditional generation via Bayesian optimization in latent space](https://nbviewer.jupyter.org/github/krasserm/bayesian-machine-learning/blob/master/variational_autoencoder_opt.ipynb). 
  Describes an approach for conditionally generating outputs with desired properties by doing Bayesian optimization in 
- latent space of variational auto-encoders. Example application implemented with Keras and GPyOpt.
+ latent space of variational autoencoders. Example application implemented with Keras and GPyOpt.

latent_variable_models_part_1.ipynb

@@ -28,7 +28,7 @@
  "cell_type": "markdown",
  "metadata": {},
  "source": [
- "# Latent variable models - part 1: Gaussian mixture models and the EM algorithm\n",
+ "# Latent variable models, part 1: Gaussian mixture models and the EM algorithm\n",
  "\n",
  "This is part 1 of a two-part series of articles about latent variable models. Part 1 covers the expectation maximization (EM) algorithm and its application to Gaussian mixture models. [Part 2](latent_variable_models_part_2.ipynb) covers approximate inference and variational autoencoders.\n",
  "\n",
@@ -711,4 +711,4 @@
  },
  "nbformat": 4,
  "nbformat_minor": 2
-}
+}

latent_variable_models_part_2.ipynb

@@ -26,7 +26,7 @@
  "cell_type": "markdown",
  "metadata": {},
  "source": [
- "# Latent variable models - part 2: Stochastic variational inference and variational autoencoders\n",
+ "# Latent variable models, part 2: Stochastic variational inference and variational autoencoders\n",
  "\n",
  "[Part 1](latent_variable_models_part_1.ipynb) of this article series introduced a latent variable model with discrete latent variables, the Gaussian mixture model (GMM), and an algorithm to fit this model to data, the EM algorithm. Part 2 covers a latent variable model with continuous latent variables for modeling more complex data, like natural images for example, and a Bayesian inference technique that can be used in conjunction with stochastic optimization algorithms.\n",
  "\n",
@@ -155,7 +155,7 @@
  "\n",
  "From the perspective of a generative model, $q(\\mathbf{t}_i \\lvert \\mathbf{x}_i, \\boldsymbol{\\phi})$ is a probabilistic *encoder* because it generates a *latent code* $\\mathbf{t}_i$ for input image $\\mathbf{x}_i$ and $p(\\mathbf{x}_i \\lvert \\mathbf{t}_i, \\boldsymbol{\\theta})$ is a probabilistic *decoder* because it generates or reconstructs an image $\\mathbf{x}_i$ from latent code $\\mathbf{t}_i$. Optimizing the variational lower bound w.r.t. parameters $\\boldsymbol{\\theta}$ and $\\boldsymbol{\\phi}$ can therefore be regarded as training a probabilistic autoencoder or *variational autoencoder* (VAE)<sup>[1]</sup>.\n",
  "\n",
- "In this context, the first term on the RHS of Eq. $(5)$ can be interpreted as expected negative *reconstruction error*. The second term is a *regularization term* that encourages the variational distribution to be close to the prior over latent variables. If the regularization term is omitted, the variational distribution would collapse to a delta function and the variational auto-encoder would degenerate to a \"usual\" deterministic autoencoder. \n",
+ "In this context, the first term on the RHS of Eq. $(5)$ can be interpreted as expected negative *reconstruction error*. The second term is a *regularization term* that encourages the variational distribution to be close to the prior over latent variables. If the regularization term is omitted, the variational distribution would collapse to a delta function and the variational autoencoder would degenerate to a \"usual\" deterministic autoencoder. \n",
  "\n",
  "### Implementation\n",
  "\n",

variational_autoencoder.ipynb

@@ -156,7 +156,7 @@
  "def neg_variational_lower_bound(x, t_decoded):\n",
  " '''\n",
  " Negative variational lower bound used as loss function\n",
- " for training the variational auto-encoder.\n",
+ " for training the variational autoencoder.\n",
  " \n",
  " Args:\n",
  " x: input images\n",
@@ -292,10 +292,10 @@
  "x_test = x_test.astype('float32') / 255.\n",
  "x_test = x_test.reshape(x_test.shape + (1,))\n",
  "\n",
- "# Compile variational auto-encoder model\n",
+ "# Compile variational autoencoder model\n",
  "vae.compile(optimizer='rmsprop', loss=neg_variational_lower_bound)\n",
  "\n",
- "# Train variational auto-encoder with MNIST images\n",
+ "# Train variational autoencoder with MNIST images\n",
  "vae.fit(x=x_train, \n",
  " y=x_train,\n",
  " epochs=25,\n",

variational_autoencoder_dfc.ipynb

@@ -4,19 +4,19 @@
  "cell_type": "markdown",
  "metadata": {},
  "source": [
- "# Deep feature consistent variational auto-encoder\n",
+ "# Deep feature consistent variational autoencoder\n",
  "\n",
  "## Introduction\n",
  "\n",
- "This article introduces the *deep feature consistent variational auto-encoder*<sup>[1]</sup> (DFC VAE) and provides a Keras implementation to demonstrate the advantages over a plain *variational auto-encoder*<sup>[2]</sup> (VAE).\n",
+ "This article introduces the *deep feature consistent variational autoencoder*<sup>[1]</sup> (DFC VAE) and provides a Keras implementation to demonstrate the advantages over a plain *variational autoencoder*<sup>[2]</sup> (VAE).\n",
  "\n",
  "A plain VAE is trained with a loss function that makes pixel-by-pixel comparisons between the original image and the reconstructured image. This often leads to generated images that are rather blurry. DFC VAEs on the other hand are trained with a loss function that first feeds the original and reconstructed image into a pre-trained convolutional neural network (CNN) to extract higher level features and then compares the these features to compute a so-called *perceptual loss*. \n",
  "\n",
  "The core idea of the perceptual loss is to seek consistency between the hidden representations of two images. Images that are perceived to be similar should also have a small perceptual loss even if they significantly differ in a pixel-by-pixel comparison (due to translation, rotation, ...). This results in generated images that look more naturally and are less blurry. The CNN used for feature extraction is called *perceptual model* in this article.\n",
  "\n",
  "### Plain VAE\n",
  "\n",
- "In a [previous article](https://nbviewer.jupyter.org/github/krasserm/bayesian-machine-learning/blob/master/variational_autoencoder.ipynb?flush_cache=true) I introduced the variational auto-encoder (VAE) and how it can be trained with a variational lower bound $\\mathcal{L}$ as optimization objective using stochastic gradient ascent methods. In context of stochastic gradient descent its negative value is used as loss function $L_{vae}$ which is a sum of a reconstruction loss $L_{rec}$ and a regularization term $L_{kl}$:\n",
+ "In a [previous article](https://nbviewer.jupyter.org/github/krasserm/bayesian-machine-learning/blob/master/variational_autoencoder.ipynb?flush_cache=true) I introduced the variational autoencoder (VAE) and how it can be trained with a variational lower bound $\\mathcal{L}$ as optimization objective using stochastic gradient ascent methods. In context of stochastic gradient descent its negative value is used as loss function $L_{vae}$ which is a sum of a reconstruction loss $L_{rec}$ and a regularization term $L_{kl}$:\n",
  "\n",
  "$$\n",
  "\\begin{align*}\n",
@@ -42,7 +42,7 @@
  "cell_type": "markdown",
  "metadata": {},
  "source": [
- "*Fig. 1: Plain variational auto-encoder*"
+ "*Fig. 1: Plain variational autoencoder*"
  ]
  },
  {
@@ -78,7 +78,7 @@
  "cell_type": "markdown",
  "metadata": {},
  "source": [
- "*Fig. 2. Deep feature consistent variational auto-encoder*"
+ "*Fig. 2. Deep feature consistent variational autoencoder*"
  ]
  },
  {
@@ -276,7 +276,7 @@
  "cell_type": "markdown",
  "metadata": {},
  "source": [
- "After loading the MNIST dataset and normalizing pixel values to interval $[0,1]$ we have now everything we need to train the two auto-encoders. This takes a few minutes per model on a GPU. The default setting however is to load the pre-trained weights for the auto-encoders instead of training them."
+ "After loading the MNIST dataset and normalizing pixel values to interval $[0,1]$ we have now everything we need to train the two autoencoders. This takes a few minutes per model on a GPU. The default setting however is to load the pre-trained weights for the autoencoders instead of training them."
  ]
  },
  {
@@ -367,17 +367,17 @@
  "from variational_autoencoder_dfc_util import plot_image_rows\n",
  "\n",
  "def encode(model, images):\n",
- " '''Encodes images with the encoder of the given auto-encoder model'''\n",
+ " '''Encodes images with the encoder of the given autoencoder model'''\n",
  " return model.get_layer('encoder').predict(images)[0]\n",
  "\n",
  "\n",
  "def decode(model, codes):\n",
- " '''Decodes latent vectors with the decoder of the given auto-encoder model'''\n",
+ " '''Decodes latent vectors with the decoder of the given autoencoder model'''\n",
  " return model.get_layer('decoder').predict(codes)\n",
  "\n",
  "\n",
  "def encode_decode(model, images):\n",
- " '''Encodes and decodes an image with the given auto-encoder model'''\n",
+ " '''Encodes and decodes an image with the given autoencoder model'''\n",
  " return decode(model, encode(model, images))\n",
  "\n",
  "\n",
@@ -497,7 +497,7 @@
  "cell_type": "markdown",
  "metadata": {},
  "source": [
- "On average, the original images have the highest Laplacian variance (highest focus or least blur) whereas the reconstructed images are more blurry. But the images reconstructed by the DFC VAE are significantly less blurry than those reconstructed by the plain VAE. The statistical significance of this difference can verified with a [t-test](https://en.wikipedia.org/wiki/Student%27s_t-test) for paired samples (the same test images are used by both auto-encoders):"
+ "On average, the original images have the highest Laplacian variance (highest focus or least blur) whereas the reconstructed images are more blurry. But the images reconstructed by the DFC VAE are significantly less blurry than those reconstructed by the plain VAE. The statistical significance of this difference can verified with a [t-test](https://en.wikipedia.org/wiki/Student%27s_t-test) for paired samples (the same test images are used by both autoencoders):"
  ]
  },
  {