Update lenet_forward_pass.ipynb given the recent changes

Author: Maksymilian Graczyk

examples/lenet_forward_pass.ipynb

@@ -1,5 +1,26 @@
 {
  "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Running a forward pass through LeNet using MNIST and Joey"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "In this example, we will construct LeNet using Joey, set it up with pretrained parameters and run a forward pass through it with test data from MNIST. The results will be compared to the PyTorch ones to confirm Joey's numerical correctness."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Firstly, let's import all the prerequisites:"
+ ]
+ },
  {
  "cell_type": "code",
  "execution_count": 1,
@@ -8,7 +29,21 @@
  "source": [
  "import torch\n",
  "import torchvision\n",
- "import torchvision.transforms as transforms"
+ "import torchvision.transforms as transforms\n",
+ "import torch.nn as nn\n",
+ "import torch.nn.functional as F\n",
+ "import torch.optim as optim\n",
+ "import matplotlib.pyplot as plt\n",
+ "import numpy as np\n",
+ "import joey as ml\n",
+ "from joey.activation import ReLU"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We'll define `imshow()` to quickly have a look at the MNIST data we'll use for the forward pass."
  ]
  },
  {
@@ -17,16 +52,22 @@
  "metadata": {},
  "outputs": [],
  "source": [
- "import matplotlib.pyplot as plt\n",
- "import numpy as np\n",
- "\n",
  "def imshow(img):\n",
  " img = img / 2 + 0.5\n",
  " npimg = img.numpy()\n",
  " plt.imshow(np.transpose(npimg, (1, 2, 0)))\n",
  " plt.show()"
  ]
  },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Before we start working with Joey, we have to download the images and convert them to NumPy arrays with `dtype=np.float64`. This is because Joey supports only NumPy arrays (rather than PyTorch tensors) and it currently works with double floating-point numbers.\n",
+ "\n",
+ "In our case, a batch will consist of 4 elements."
+ ]
+ },
  {
  "cell_type": "code",
  "execution_count": 3,
@@ -41,22 +82,22 @@
  "testloader = torch.utils.data.DataLoader(testset, batch_size=4, shuffle=False, num_workers=2)\n",
  "\n",
  "classes = ('0', '1', '2', '3', '4', '5', '6', '7', '8', '9')\n",
- "dataiter = iter(testloader)"
+ "dataiter = iter(testloader)\n",
+ "\n",
+ "images, labels = dataiter.next()\n",
+ "input_data = images.double().numpy()"
  ]
  },
  {
- "cell_type": "code",
- "execution_count": 4,
+ "cell_type": "markdown",
  "metadata": {},
- "outputs": [],
  "source": [
- "images, labels = dataiter.next()\n",
- "input_data = images.numpy()"
+ "Let's have a look at what we've downloaded:"
  ]
  },
  {
  "cell_type": "code",
- "execution_count": 5,
+ "execution_count": 4,
  "metadata": {},
  "outputs": [
  {
@@ -76,16 +117,21 @@
  "imshow(torchvision.utils.make_grid(images))"
  ]
  },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now, we'll define `forward_pass()`. It creates LeNet using the `Net` class in Joey along with appropriate layer classes (here: `Conv`, `MaxPooling`, `Flat` and `FullyConnected`). Afterwards, by accessing the `kernel` and `bias` properties of each relevant layer, it inserts the pretrained weights saved in `.npy` files inside `resources/`.\n",
+ "\n",
+ "Note that we have to disable a strict stride check in `layer4`. If we didn't do that, we would get an error saying the stride is incompatible with the provided kernel and input sizes."
+ ]
+ },
  {
  "cell_type": "code",
- "execution_count": 6,
+ "execution_count": 5,
  "metadata": {},
  "outputs": [],
  "source": [
- "import joey as ml\n",
- "import numpy as np\n",
- "from joey.activation import ReLU\n",
- "\n",
  "def forward_pass(input_data):\n",
  " parameters = get_parameters()\n",
  " \n",
@@ -122,22 +168,33 @@
  " # Flattening layer necessary between layer 4 and 5\n",
  " layer_flat = ml.Flat(input_size=(batch_size, 16, 6, 6))\n",
  " \n",
- " current_data = input_data\n",
- " current_data = layer1.execute(current_data, parameters[1], parameters[0])\n",
- " current_data = layer2.execute(current_data)\n",
- " current_data = layer3.execute(current_data, parameters[3], parameters[2])\n",
- " current_data = layer4.execute(current_data)\n",
+ " layers = [layer1, layer2, layer3, layer4,\n",
+ " layer_flat, layer5, layer6, layer7]\n",
+ " \n",
+ " net = ml.Net(layers)\n",
+ " \n",
+ " # Setting up the pretrained parameters\n",
+ " layer1.kernel.data[:] = parameters[0]\n",
+ " layer1.bias.data[:] = parameters[1]\n",
+ " \n",
+ " layer3.kernel.data[:] = parameters[2]\n",
+ " layer3.bias.data[:] = parameters[3]\n",
  " \n",
- " current_data = layer_flat.execute(current_data)\n",
+ " layer5.kernel.data[:] = parameters[4]\n",
+ " layer5.bias.data[:] = parameters[5]\n",
  " \n",
- " current_data = layer5.execute(current_data, parameters[5], parameters[4])\n",
- " current_data = layer6.execute(current_data, parameters[7], parameters[6])\n",
- " current_data = layer7.execute(current_data, parameters[9], parameters[8])\n",
+ " layer6.kernel.data[:] = parameters[6]\n",
+ " layer6.bias.data[:] = parameters[7]\n",
  " \n",
- " return current_data\n",
+ " layer7.kernel.data[:] = parameters[8]\n",
+ " layer7.bias.data[:] = parameters[9]\n",
+ " \n",
+ " net.forward(input_data)\n",
+ " \n",
+ " return (layer1, layer2, layer3, layer4, layer5, layer6, layer7)\n",
  "\n",
  "def get_parameters():\n",
- " # The LeNet trained parameters are stored in the following files:\n",
+ " # The LeNet pretrained parameters are stored in the following files:\n",
  " # 1.npy: layer 1 weights\n",
  " # 2.npy: layer 1 biases\n",
  " # 3.npy: layer 3 weights\n",
@@ -159,9 +216,16 @@
  " return parameters"
  ]
  },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "At this point, we're ready to run the forward pass!"
+ ]
+ },
  {
  "cell_type": "code",
- "execution_count": 7,
+ "execution_count": 6,
  "metadata": {},
  "outputs": [
  {
@@ -170,24 +234,24 @@
  "text": [
  "/home/maksymilian/Desktop/UROP/devito/devito/types/grid.py:206: RuntimeWarning: divide by zero encountered in true_divide\n",
  " spacing = (np.array(self.extent) / (np.array(self.shape) - 1)).astype(self.dtype)\n",
- "Operator `Kernel` run in 0.01 s\n",
- "Operator `Kernel` run in 0.01 s\n",
- "Operator `Kernel` run in 0.01 s\n",
- "Operator `Kernel` run in 0.01 s\n",
- "Operator `Kernel` run in 0.01 s\n",
- "Operator `Kernel` run in 0.01 s\n",
- "Operator `Kernel` run in 0.01 s\n",
  "Operator `Kernel` run in 0.01 s\n"
  ]
  }
  ],
  "source": [
- "results = forward_pass(input_data)"
+ "layer1, layer2, layer3, layer4, layer5, layer6, layer7 = forward_pass(input_data)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "After the pass is finished, we can access its output by checking the `result` property of the last layer."
  ]
  },
  {
  "cell_type": "code",
- "execution_count": 8,
+ "execution_count": 7,
  "metadata": {},
  "outputs": [
  {
@@ -208,7 +272,49 @@
  }
  ],
  "source": [
- "print(results)"
+ "output = layer7.result.data\n",
+ "print(output)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The results look promising: for each batch element (arranged in columns rather than rows), the highest number corresponds to the expected class, i.e. '7' has been recognised as 7, '2' has been recognised as 2, '1' has been recognised as 1 and '0' has been recognised as 0.\n",
+ "\n",
+ "For reference, we'll construct the same network with the same weights in PyTorch, run the pass there and compare the outputs."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "class Net(nn.Module):\n",
+ " def __init__(self):\n",
+ " super(Net, self).__init__()\n",
+ " self.conv1 = nn.Conv2d(1, 6, 3)\n",
+ " self.conv2 = nn.Conv2d(6, 16, 3)\n",
+ " self.fc1 = nn.Linear(16 * 6 * 6, 120)\n",
+ " self.fc2 = nn.Linear(120, 84)\n",
+ " self.fc3 = nn.Linear(84, 10)\n",
+ " \n",
+ " def forward(self, x):\n",
+ " x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))\n",
+ " x = F.max_pool2d(F.relu(self.conv2(x)), 2)\n",
+ " x = x.view(-1, self.num_flat_features(x))\n",
+ " x = F.relu(self.fc1(x))\n",
+ " x = F.relu(self.fc2(x))\n",
+ " x = self.fc3(x)\n",
+ " return x\n",
+ " \n",
+ " def num_flat_features(self, x):\n",
+ " size = x.size()[1:]\n",
+ " num_features = 1\n",
+ " for s in size:\n",
+ " num_features *= s\n",
+ " return num_features"
  ]
  },
  {
@@ -217,17 +323,29 @@
  "metadata": {},
  "outputs": [],
  "source": [
- "expected = np.array([[-1.2509323 , 2.4017086 , -2.9189475 , 11.402614 ],\n",
- " [-2.0739274 , 3.711194 , 10.299156 , -3.8691325 ],\n",
- " [ 1.7185768 , 11.983461 , 0.7847577 , -0.8381885 ],\n",
- " [ 2.7290256 , 1.5788821 , -2.2999125 , -2.1093633 ],\n",
- " [-3.447302 , -0.9786217 , 0.7426771 , -2.761261 ],\n",
- " [-2.2462513 , -6.905971 , -2.5677016 , 0.6907152 ],\n",
- " [-9.817931 , -1.3155346 , -2.7154455 , 1.1705266 ],\n",
- " [11.809888 , -2.7028327 , 0.54783845, 1.0049481 ],\n",
- " [-1.0047406 , -2.4807127 , -1.013465 , -1.2820265 ],\n",
- " [ 4.6835623 , -6.3834734 , -2.2608764 , -0.76408434]])\n",
- "error = abs(results - expected) / expected"
+ "net = Net()\n",
+ "net.double()\n",
+ "\n",
+ "with torch.no_grad():\n",
+ " net.conv1.weight[:] = torch.from_numpy(layer1.kernel.data)\n",
+ " net.conv1.bias[:] = torch.from_numpy(layer1.bias.data)\n",
+ " net.conv2.weight[:] = torch.from_numpy(layer3.kernel.data)\n",
+ " net.conv2.bias[:] = torch.from_numpy(layer3.bias.data)\n",
+ " net.fc1.weight[:] = torch.from_numpy(layer5.kernel.data)\n",
+ " net.fc1.bias[:] = torch.from_numpy(layer5.bias.data)\n",
+ " net.fc2.weight[:] = torch.from_numpy(layer6.kernel.data)\n",
+ " net.fc2.bias[:] = torch.from_numpy(layer6.bias.data)\n",
+ " net.fc3.weight[:] = torch.from_numpy(layer7.kernel.data)\n",
+ " net.fc3.bias[:] = torch.from_numpy(layer7.bias.data)\n",
+ "\n",
+ "pytorch_output = np.transpose(net(images.double()).detach().numpy())"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "After creating and running the network in PyTorch, we'll calculate a relative error matrix as shown below. The maximum value in that matrix will be obtained as well."
  ]
  },
  {
@@ -239,23 +357,32 @@
  "name": "stdout",
  "output_type": "stream",
  "text": [
- "[[-3.63663429e-08 4.40914511e-07 -3.72550049e-08 3.25849137e-07]\n",
- " [-1.05593081e-07 4.90411682e-08 5.72853023e-08 -2.79713692e-07]\n",
- " [ 1.45904743e-07 2.85106639e-08 4.09210438e-08 -5.01416183e-07]\n",
- " [ 1.20755307e-07 4.73338244e-07 -2.46663913e-07 -1.33781978e-07]\n",
- " [-1.13912282e-07 -1.40684427e-07 1.58951535e-07 -3.16022361e-07]\n",
- " [-2.53501737e-08 -8.97095633e-08 -2.15097806e-07 6.65195837e-07]\n",
- " [-8.21189975e-08 -2.44784091e-07 -4.71435314e-07 3.77461859e-07]\n",
- " [ 3.91327608e-08 -4.14888517e-08 3.43611389e-07 5.78272754e-08]\n",
- " [-7.21504199e-07 -1.64031238e-07 -1.47184007e-07 -3.07741901e-07]\n",
- " [ 1.71215680e-07 -8.06319813e-08 -3.03396411e-07 -1.81879287e-06]]\n",
- "6.651958371252912e-07\n"
+ "[[1.77503288e-16 3.69811230e-16 3.04280379e-16 0.00000000e+00]\n",
+ " [8.56518243e-16 5.98310452e-16 1.72475952e-16 0.00000000e+00]\n",
+ " [1.42122890e-15 1.48234044e-16 4.24420039e-16 2.11928192e-15]\n",
+ " [4.88184424e-16 9.84437976e-16 5.79268976e-16 2.10532377e-16]\n",
+ " [1.28822268e-16 4.53790543e-16 4.48468063e-16 3.21656917e-16]\n",
+ " [3.95404734e-16 2.57220454e-16 5.18855985e-16 1.12514772e-15]\n",
+ " [1.80929841e-16 1.85665208e-15 1.63541857e-16 1.89696406e-16]\n",
+ " [1.50412669e-16 4.92915335e-16 3.03982673e-15 4.41902657e-16]\n",
+ " [2.20996787e-16 1.07410088e-15 8.76378119e-16 1.73198086e-16]\n",
+ " [3.79274668e-16 4.17411542e-16 3.92847079e-16 7.26506870e-16]]\n",
+ "3.0398267312380578e-15\n"
  ]
  }
  ],
  "source": [
+ "error = abs(output - pytorch_output) / abs(pytorch_output)\n",
+ "\n",
  "print(error)\n",
- "print(np.amax(error))"
+ "print(np.nanmax(error))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "As we can see, the maximum error is low enough (given the floating-point calculation accuracy) for the Joey results to be considered numerically correct."
  ]
  }
  ],