final fixes e2e tutorial

Author: adshar

AirSimE2EDeepLearning/DataExplorationAndPreparation.ipynb

@@ -11,13 +11,13 @@
  "Our goal is to train a deep learning model that can make steering angle predictions based on an input that comprises of camera images and the vehicle's last known state. In this notebook, we will prepare the data for our end-to-end deep learning model. Along the way, we will also make some useful observations about the dataset that will aid us when it comes time to train the model. \n",
  "\n",
  "\n",
- "## What is End-to-End Deep Learning?\n",
+ "## What is end-to-end deep learning?\n",
  "\n",
- "End-to-end deep learning is a modeling strategy that is a response to the success of deep neural networks. Unlike traditional methods, this strategy is not built on feature engineering. Instead, it leverages the power of deep neural networks, along with recent hardware advances (GPUs, FPGAs etc.) to harness the incredible potential of large amounts of data. It is closer to a human-like learning approach than traditional ML as it lets a neural network map raw input to direct outputs. A big downside to this approach is that it requires a very large amount of training data which makes it unsuitable for many common applications. Since simulators can (potentially) generate infinite amounts of data, they are a perfect data source for end-to-end deep learning algorithms. If you wish to learn more, [this video](https://www.coursera.org/learn/machine-learning-projects/lecture/k0Klk/what-is-end-to-end-deep-learning) by Andrew Ng provides a nice overview of the topic.\n",
+ "End-to-end deep learning is a modeling strategy that is a response to the success of deep neural networks. Unlike traditional methods, this strategy is not built on feature engineering. Instead, it leverages the power of deep neural networks, along with recent hardware advances (GPUs, FPGAs etc.) to harness the incredible potential of large amounts of data. It is closer to a human-like learning approach than traditional ML as it lets a neural network map raw input to direct output. A big downside to this approach is that it requires a very large amount of training data which makes it unsuitable for many common applications. Since simulators can (potentially) generate data in infinite amounts, they are a perfect data source for end-to-end deep learning algorithms. If you wish to learn more, [this video](https://www.coursera.org/learn/machine-learning-projects/lecture/k0Klk/what-is-end-to-end-deep-learning) by Andrew Ng provides a nice overview of the topic.\n",
  "\n",
- "Autonomous driving is a field that can highly benefit from the power of end-to-end deep learning. In order to achieve SAE Level 4 Autonomy, cars need to be trained on copious amounts of data (it is not uncommon for car manufacturers to collect hundreds of petabytes of data every week), something that is virtually impossible without a simulator. \n",
+ "Autonomous driving is a field that can highly benefit from the power of end-to-end deep learning. In order to achieve SAE Level 4 or 5 Autonomy, cars need to be trained on copious amounts of data (it is not uncommon for car manufacturers to collect hundreds of petabytes of data every week), something that is virtually impossible without a simulator. \n",
  "\n",
- "With photo-realistic simulators like [AirSim](https://github.com/Microsoft/AirSim), it is now possible to collect a large amount of data to train your autonomous driving models without having to use an actual car. These models can then be fine tuned using a comparably lesser amount of real-world data and used on actual cars. This technique is called Behavioral Cloning. In this tutorial, you will train a model to learn how to steer a car through a portion of the Landscape map in AirSim using only one of the front facing webcams as visual input. Our strategy will be to perform some basic data analysis to get a feel for the dataset, and then train an end-to-end deep learning model to predict the correct steering control signals (a.k.a. \"steering angle\") given a frame from the webcam, and the car's current state parameters (speed, steering angle, throttle etc.).\n",
+ "With photo-realistic simulators like [AirSim](https://github.com/Microsoft/AirSim), it is now possible to collect a large amount of data to train your autonomous driving models without having to use an actual car. These models can then be fine tuned using a comparably lesser amount of real-world data and used on actual cars. This technique is called Behavioral Cloning. In this tutorial, you will train a model to learn how to steer a car through a portion of the Landscape map in AirSim using only one of the front facing webcams on the car as visual input. Our strategy will be to perform some basic data analysis to get a feel for the dataset, and then train an end-to-end deep learning model to predict the correct driving control signal (in this case the steering angle) given a frame from the webcam, and the car's current state parameters (speed, steering angle, throttle etc.).\n",
  "\n",
  "Before you begin, please make sure you have the dataset for the tutorial downloaded. If you missed the instructions in the readme file, [you can download the dataset from here](https://aka.ms/AirSimTutorialDataset).\n",
  "\n",
@@ -62,13 +62,14 @@
  "cell_type": "markdown",
  "metadata": {},
  "source": [
- "First, let's take a look at the raw data. There are two parts to the dataset - the images and the .tsv file. First, let us read one of the .tsv files."
+ "Let's take a look at the raw data. There are two parts to the dataset - the images and the .tsv file. First, let us read one of the .tsv files."
  ]
  },
  {
  "cell_type": "code",
  "execution_count": 2,
  "metadata": {
+ "collapsed": false,
  "scrolled": true
  },
  "outputs": [
@@ -181,13 +182,14 @@
  "cell_type": "markdown",
  "metadata": {},
  "source": [
- "This dataset contains our label, the steering angle. It also has the name of the image taken at the time the steering angle was recorded. Let's look at a sample image - 'img_0.png' inside the 'normal_1' folder (more on folder naming later)."
+ "This dataset contains our label, the steering angle. It also has the name of the image taken at the time the steering angle was recorded. Let's look at a sample image - 'img_0.png' inside the 'normal_1' folder (more on our folder naming style later)."
  ]
  },
  {
  "cell_type": "code",
  "execution_count": 3,
  "metadata": {
+ "collapsed": false,
  "scrolled": true
  },
  "outputs": [
@@ -220,7 +222,9 @@
  {
  "cell_type": "code",
  "execution_count": 4,
- "metadata": {},
+ "metadata": {
+ "collapsed": false
+ },
  "outputs": [
  {
  "data": {
@@ -252,7 +256,7 @@
  "cell_type": "markdown",
  "metadata": {},
  "source": [
- "**Extracting this ROI will both reduce the training time and the amount of data needed to train the model**. It will also prevent the model from getting confused by focusing on irrelevant features in the environment (e.g. clouds, birds, etc)\n",
+ "**Extracting this ROI will both reduce the training time and the amount of data needed to train the model**. It will also prevent the model from getting confused by focusing on irrelevant features in the environment (e.g. mountains, trees, etc)\n",
  "\n",
  "Another observation we can make is that **the dataset exhibits a vertical flip tolerance**. That is, we get a valid data point if we flip the image around the Y axis if we also flip the sign of the steering angle. This is important as it effectively doubles the number of data points we have available. \n",
  "\n",
@@ -274,7 +278,9 @@
  {
  "cell_type": "code",
  "execution_count": 5,
- "metadata": {},
+ "metadata": {
+ "collapsed": false
+ },
  "outputs": [
  {
  "name": "stdout",
@@ -415,13 +421,15 @@
  "cell_type": "markdown",
  "metadata": {},
  "source": [
- "Let us first address the naming of the dataset folders. You will notice that we have two types of folders in our dataset - 'normal', and 'swerve'. These names refer to two different driving strategies. Let's begin by attempting to get an understanding of the differences between these two styles of driving. First, we'll plot a portion of datapoints from each of the driving styles against each other."
+ "Let us now address the naming of the dataset folders. You will notice that we have two types of folders in our dataset - 'normal', and 'swerve'. These names refer to two different driving strategies. Let's begin by attempting to get an understanding of the differences between these two styles of driving. First, we'll plot a portion of datapoints from each of the driving styles against each other."
  ]
  },
  {
  "cell_type": "code",
  "execution_count": 6,
- "metadata": {},
+ "metadata": {
+ "collapsed": false
+ },
  "outputs": [
  {
  "data": {
@@ -471,7 +479,9 @@
  {
  "cell_type": "code",
  "execution_count": 7,
- "metadata": {},
+ "metadata": {
+ "collapsed": false
+ },
  "outputs": [
  {
  "data": {
@@ -510,15 +520,17 @@
  "So, roughly a quarter of the data points are collected with the swerving driving strategy, and the rest are collected with the normal strategy. We also see that we have almost 47,000 data points to work with. This is nearly not enough data, hence our network cannot be too deep. \n",
  "\n",
  "> **Thought Exercise 0.4:**\n",
- "Like many things in the field of Machine Learning, the ideal blend of number of datapoints in each category here is something that is problem specific, and can only be optimized by trial and error. Can you find a split that works better than ours?\n",
+ "Like many things in the field of Machine Learning, the ideal ratio of number of datapoints in each category here is something that is problem specific, and can only be optimized by trial and error. Can you find a split that works better than ours?\n",
  "\n",
  "Let's see what the distribution of labels looks like for the two strategies."
  ]
  },
  {
  "cell_type": "code",
  "execution_count": 8,
- "metadata": {},
+ "metadata": {
+ "collapsed": false
+ },
  "outputs": [
  {
  "data": {
@@ -570,18 +582,19 @@
  "\n",
  "The code for cooking the dataset is straightforward, but long. When it terminates, the final dataset will have 4 parts:\n",
  "\n",
- "* image: a numpy array containing the image data\n",
- "* previous_state: a numpy array containing the last known state of the car. This is a (steering, throttle, brake, speed) tuple\n",
- "* label: a numpy array containing the steering angles that we wish to predict (normalized on the range -1..1)\n",
- "* metadata: a numpy array containing metadata about the files (which folder they came from, etc)\n",
+ "* **image**: a numpy array containing the image data\n",
+ "* **previous_state**: a numpy array containing the last known state of the car. This is a (steering, throttle, brake, speed) tuple\n",
+ "* **label**: a numpy array containing the steering angles that we wish to predict (normalized on the range -1..1)\n",
+ "* **metadata**: a numpy array containing metadata about the files (which folder they came from, etc)\n",
  "\n",
- "The processing may take some time. We will also divide the datasets into train/test/validation datasets."
+ "The processing may take some time. We will also combine all the datasets into one and then split it into train/test/validation datasets."
  ]
  },
  {
  "cell_type": "code",
  "execution_count": 9,
  "metadata": {
+ "collapsed": false,
  "scrolled": true
  },
  "outputs": [

AirSimE2EDeepLearning/README.md

@@ -34,7 +34,7 @@ If you have never worked with Python notebooks before, we highly recommend [chec
 
 ### Background needed
 
-At the very least, you need to be familiar with the basics neural networks. You are not required to know advanced concepts like LSTMs or Reinforcement Learning but you should know how Convolutional Networks work. A really good starting point to get a strong background in a very short amount of time is [this great book](http://neuralnetworksanddeeplearning.com/) written by Michael Nielsen. It is free, very short and available online. It can provide you a solid foundation in less than a week's time.
+You should be familiar with the basics of neural networks and deep learning. You are not required to know advanced concepts like LSTMs or Reinforcement Learning but you should know how Convolutional Neural Networks work. A really good starting point to get a strong background in a short amount of time is [this highly recommended book on the topic](http://neuralnetworksanddeeplearning.com/) written by Michael Nielsen. It is free, very short and available online. It can provide you a solid foundation in less than a week's time.
 
 You should also be comfortable with Python. At the very least, you should be able to read and understand code written in Python. 
 
@@ -44,10 +44,17 @@ You should also be comfortable with Python. At the very least, you should be abl
 2. [Install CNTK](https://docs.microsoft.com/en-us/cognitive-toolkit/Setup-CNTK-on-your-machine) or [install Tensorflow](https://www.tensorflow.org/install/install_windows)
 3. [Install h5py](http://docs.h5py.org/en/latest/build.html)
 4. [Install Keras](https://keras.io/#installation) and [configure the Keras backend](https://keras.io/backend/) to work with TensorFlow (default) or CNTK.
+5. [Install AzCopy](https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azcopy). Be sure to add the location for the AzCopy executable to your system path.
 
 ### Simulator Package
 
-We have created a standalone build of the AirSim simulation environment for the tutorials in this cookbook. [You can download the build package from here](https://aka.ms/ADCookbookAirSimPackage). After downloading the package, unzip it and run the **ADCookbook_Airsim_Mountain.bat** file to start the simulator.
+We have created a standalone build of the AirSim simulation environment for the tutorials in this cookbook. [You can download the build package from here](https://airsimtutorialdataset.blob.core.windows.net/e2edl/AD_Cookbook_AirSim.7z). Consider using [AzCopy](https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azcopy), as the file size is large. After downloading the package, unzip it and run the PowerShell command
+
+`
+.\AD_Cookbook_Start_AirSim.ps1 landscape
+`
+
+to start the simulator in the landscape environment.
 
 ### Hardware
 

AirSimE2EDeepLearning/TestModel.ipynb

@@ -41,7 +41,9 @@
  {
  "cell_type": "code",
  "execution_count": 2,
- "metadata": {},
+ "metadata": {
+ "collapsed": false
+ },
  "outputs": [
  {
  "name": "stdout",
@@ -153,7 +155,7 @@
  "> ** Thought Exercise 2.2**:\n",
  "The car seems to crash when it tries to climb one of those hills. Can you think of a reason why? How can you fix this? (Hint: You might want to take a look at what the car is seeing when it is making that ascent)\n",
  "\n",
- "AirSim opens up a world of possibilities. There is no limit to the new things you can try as you train even more complex models and use other learning techniques. Here are a few immediate things you could try that might require modifying some of the code provided in this tutorials (including the helper files) but won't require modifying any Unreal assets.\n",
+ "AirSim opens up a world of possibilities. There is no limit to the new things you can try as you train even more complex models and use other learning techniques. Here are a few immediate things you could try that might require modifying some of the code provided in this tutorial (including the helper files) but won't require modifying any Unreal assets.\n",
  "\n",
  "> ** Exploratory Idea 2.1**:\n",
  "If you have a background in Machine Learning, you might have asked the question: why did we train and test in the same environment? Isn't that overfitting? Well, you can make arguments on both sides. While using the same environment for both training and testing might seem like you are overfitting to that environment, it can also be seen as drawing examples from the same probability distribution. The data used for training and testing is not the same, even though it is coming from the same distribution. So that brings us to the question: how will this model fare in a different environment, one it hasn't seen before? \n",
@@ -167,7 +169,7 @@
  "The model currently views a single image and a single state for each prediction. However, we have access to historical data. Can we extend the model to make predictions using the previous N images and states (e.g. given the past 3 images and past 3 states, predict the next steering angle)? (Hint: This will possibly require you to use recurrent neural network techniques)\n",
  "\n",
  "> ** Exploratory Idea 2.4**:\n",
- "AirSim is much more than the dataset we provided you. For starters, we only used one camera and used it only in RGB mode. AirSim lets you collect data in depth view, segmentation view, surface normal view etc for each of the cameras available. So you can potentially have 20 different images (for 5 cameras operating in all 4 modes) for each instance (we only used 1 image here). How can combining all this information help us improve the model we just trained?"
+ "AirSim is a lot more than the dataset we provided you. For starters, we only used one camera and used it only in RGB mode. AirSim lets you collect data in depth view, segmentation view, surface normal view etc for each of the cameras available. So you can potentially have 20 different images (for 5 cameras operating in all 4 modes) for each instance (we only used 1 image here). How can combining all this information help us improve the model we just trained?"
  ]
  }
  ],
@@ -187,7 +189,7 @@
  "name": "python",
  "nbconvert_exporter": "python",
  "pygments_lexer": "ipython3",
- "version": "3.5.3"
+ "version": "3.6.0"
  }
  },
  "nbformat": 4,