Adding new Neo folder and Region check (aws#810)

* Add Neo folder and move Neo notebook under it * Update README * Update mxnet_mnist_neo.ipynb with Evidence * add compile section in mnist notebook for mxnet and tensorflow * clean serializer function in neo tensorflow mnist * Add Neo Region Check for tensorflow_mnist and mxnet_mnist

Author: Xingyu Zhou

README.md

@@ -109,6 +109,15 @@ These examples that showcase unique functionality available in Amazon SageMaker.
 - [Inference Pipeline with SparkML and BlazingText](advanced_functionality/inference_pipeline_sparkml_blazingtext_dbpedia) shows how to deploy an Inference Pipeline with SparkML for data pre-processing and BlazingText for training on the DBPedia dataset. The pre-processing code is written once and used between training and inference.
 - [Experiment Management Capabilities with Search](advanced_functionality/search) shows how to organize Training Jobs into projects, and track relationships between Models, Endpoints, and Training Jobs.
 
+### Amazon SageMaker Neo Compilation Jobs
+
+These examples provide you an introduction to how to use Neo to optimizes deep learning model
+
+- [Image Classification](sagemaker_neo_compilation_jobs/imageclassification_caltech) Adapts form [image classification](introduction_to_amazon_algorithms/imageclassification_caltech) including Neo API and comparsion between the baseline
+- [MNIST with MXNet](sagemaker_neo_compilation_jobs/mxnet_mnist) Adapts form [mxnet mnist](sagemaker-python-sdk/mxnet_mnist) including Neo API and comparsion between the baseline
+- [Deploying pre-trained PyTorch vision models](sagemaker_neo_compilation_jobs/pytorch_torchvision) shows how to use Amazon SageMaker Neo to compile and optimize pre-trained PyTorch models from TorchVision.
+- [Distributed TensorFlow](sagemaker_neo_compilation_jobs/tensorflow_distributed_mnist) Adapts form [tensorflow mnist](sagemaker-python-sdk/tensorflow_distributed_mnist) including Neo API and comparsion between the baseline
+- [Predicting Customer Churn](sagemaker_neo_compilation_jobs/xgboost_customer_churn) Adapts form [xgboost customer churn](introduction_to_applying_machine_learning/xgboost_customer_churn) including Neo API and comparsion between the baseline
 
 ### Amazon SageMaker Pre-Built Framework Containers and the Python SDK
 

advanced_functionality/README.md

@@ -19,4 +19,3 @@ These examples that showcase unique functionality available in Amazon SageMaker.
 - [Inference Pipeline with SparkML and BlazingText](inference_pipeline_sparkml_blazingtext_dbpedia) shows how to deploy an Inference Pipeline with SparkML for data pre-processing and BlazingText for training on the DBPedia dataset. The pre-processing code is written once and used between training and inference.
 - [Creating Algorithm and Model Package - Listing on AWS Marketplace](creating_marketplace_products) provides a detailed walkthrough on how to package a scikit learn algorithm to create SageMaker Algorithm and SageMaker Model Package entities that can be used with the enhanced SageMaker Train/Transform/Hosting/Tuning APIs and listed on AWS Marketplace.
 - [Using Algorithm and Model Packages - From AWS Marketplace](using_marketplace_products) provides a detailed walkthrough on how to use Algorithm and Model Package entities with the enhanced SageMaker Train/Transform/Hosting/Tuning APIs by choosing a canonical product listed on AWS Marketplace.
-- [Deploying pre-trained PyTorch vision models with Amazon SageMaker Neo](pytorch_torchvision_neo) shows how to use Amazon SageMaker Neo to compile and optimize pre-trained PyTorch models from TorchVision.

introduction_to_amazon_algorithms/imageclassification_caltech/README.md

@@ -10,9 +10,6 @@ This notebook `Imageclassification-lst-format.ipynb` demos an end-2-end system f
 ### SageMaker Image classification full training highlevel
 This notebook `ImageClassification-fulltraining-highlevel.ipynb` is similar to the `ImageClassification-fulltraining.ipynb` but using Sagemaker high-level APIs
 
-### SageMaker Image classification full training highlevel with Neo
-This notebook `ImageClassification-fulltraining-highlevel-neo.ipynb` is similar to the `ImageClassification-fulltraining.ipynb` but using Sagemaker high-level APIs(including Neo)
-
 ### SageMaker Image classification transfer learning highlevel
 This notebook `Imageclassification-transfer-learning-highlevel.ipynb` is similar to the `ImageClassification-transfer-learning.ipynb` but using Sagemaker high-level APIs
 

sagemaker-python-sdk/mxnet_mnist/mxnet_mnist.ipynb

@@ -128,6 +128,35 @@
  "mnist_estimator.fit({'train': train_data_location, 'test': test_data_location})"
  ]
  },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Opimtize your model with Neo API\n",
+ "Neo API allows to optimize our model for a specific hardware type. When calling compile_model() function, we specify the target instance family (C5) as well as the S3 bucket to which the compiled model would be stored.\n",
+ "\n",
+ "#### Important. If the following command result in a permission error, scroll up and locate the value of execution role returned by get_execution_role(). The role must have access to the S3 bucket specified in output_path."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "neo_optimize = False\n",
+ "compiled_model = mnist_estimator\n",
+ "if mnist_estimator.create_model().check_neo_region(boto3.Session().region_name) is False:\n",
+ " print('Neo is not currently supported in', boto3.Session().region_name)\n",
+ "else:\n",
+ " output_path = '/'.join(mnist_estimator.output_path.split('/')[:-1])\n",
+ " neo_optimize = True\n",
+ " compiled_model = mnist_estimator.compile_model(target_instance_family='ml_m4', \n",
+ " input_shape={'data':[1, 784]},\n",
+ " role=role,\n",
+ " output_path=output_path)"
+ ]
+ },
  {
  "cell_type": "markdown",
  "metadata": {},
@@ -147,10 +176,29 @@
  "source": [
  "%%time\n",
  "\n",
- "predictor = mnist_estimator.deploy(initial_instance_count=1,\n",
+ "predictor = compiled_model.deploy(initial_instance_count=1,\n",
  " instance_type='ml.m4.xlarge')"
  ]
  },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import io\n",
+ "import numpy as np\n",
+ "def numpy_bytes_serializer(data):\n",
+ " f = io.BytesIO()\n",
+ " np.save(f, data)\n",
+ " f.seek(0)\n",
+ " return f.read()\n",
+ "\n",
+ "if neo_optimize is True:\n",
+ " predictor.content_type = 'application/vnd+python.numpy+binary'\n",
+ " predictor.serializer = numpy_bytes_serializer"
+ ]
+ },
  {
  "cell_type": "markdown",
  "metadata": {},
@@ -193,9 +241,11 @@
  "source": [
  "response = predictor.predict(data)\n",
  "print('Raw prediction result:')\n",
+ "if neo_optimize is False:\n",
+ " response = response[0]\n",
  "print(response)\n",
  "\n",
- "labeled_predictions = list(zip(range(10), response[0]))\n",
+ "labeled_predictions = list(zip(range(10), response))\n",
  "print('Labeled predictions: ')\n",
  "print(labeled_predictions)\n",
  "\n",

sagemaker-python-sdk/tensorflow_distributed_mnist/mnist.py

@@ -123,21 +123,15 @@ def _input_fn(training_dir, training_filename, batch_size=100):
 def neo_preprocess(payload, content_type):
  import logging
  import numpy as np
- import PIL.Image # Training container doesn't have this package
  import io
 
  logging.info('Invoking user-defined pre-processing function')
 
- if content_type != 'application/x-image':
- raise RuntimeError('Content type must be application/x-image')
+ if content_type != 'application/x-image' and content_type != 'application/vnd+python.numpy+binary':
+ raise RuntimeError('Content type must be application/x-image or application/vnd+python.numpy+binary')
 
  f = io.BytesIO(payload)
- # Load image and convert to greyscale space
- image = PIL.Image.open(f).convert('L')
- # Resize
- image = np.asarray(image.resize((28, 28)))
- # Reshape
- image = image.reshape((1,-1)).astype('float32')
+ image = np.load(f)*255
 
  return image
 

sagemaker-python-sdk/tensorflow_distributed_mnist/tensorflow_distributed_mnist.ipynb

@@ -167,6 +167,35 @@
  "In the end of the training, the training job will generate a saved model for TF serving."
  ]
  },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Compiling the model\n",
+ "The input_shape is the definition for the model's input tensor and output_path is where the compiled model will be stored in S3.\n",
+ "#### Important. If the following command result in a permission error, scroll up and locate the value of execution role returned by get_execution_role(). The role must have access to the S3 bucket specified in output_path."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import boto3\n",
+ "neo_optimize = False\n",
+ "optimized_estimator = mnist_estimator\n",
+ "if mnist_estimator.create_model().check_neo_region(boto3.Session().region_name) is False:\n",
+ " print('Neo is not currently supported in', boto3.Session().region_name)\n",
+ "else:\n",
+ " output_path = '/'.join(mnist_estimator.output_path.split('/')[:-1])\n",
+ " neo_optimize = True\n",
+ " optimized_estimator = mnist_estimator.compile_model(target_instance_family='ml_m4', \n",
+ " input_shape={'data':[1, 784]}, # Batch size 1, 3 channels, 224x224 Images.\n",
+ " output_path=output_path,\n",
+ " framework='tensorflow', framework_version='1.11.0')"
+ ]
+ },
  {
  "cell_type": "markdown",
  "metadata": {
@@ -186,10 +215,26 @@
  },
  "outputs": [],
  "source": [
- "mnist_predictor = mnist_estimator.deploy(initial_instance_count=1,\n",
+ "mnist_predictor = optimized_estimator.deploy(initial_instance_count=1,\n",
  " instance_type='ml.m4.xlarge')"
  ]
  },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def numpy_bytes_serializer(data):\n",
+ " f = io.BytesIO()\n",
+ " np.save(f, data)\n",
+ " f.seek(0)\n",
+ " return f.read()\n",
+ "if neo_optimize is True:\n",
+ " mnist_predictor.content_type = 'application/vnd+python.numpy+binary'\n",
+ " mnist_predictor.serializer = numpy_bytes_serializer"
+ ]
+ },
  {
  "cell_type": "markdown",
  "metadata": {},
@@ -205,20 +250,25 @@
  },
  "outputs": [],
  "source": [
+ "import io\n",
  "import numpy as np\n",
  "from tensorflow.examples.tutorials.mnist import input_data\n",
  "\n",
  "mnist = input_data.read_data_sets(\"/tmp/data/\", one_hot=True)\n",
  "\n",
  "for i in range(10):\n",
- " data = mnist.test.images[i].tolist()\n",
- " tensor_proto = tf.make_tensor_proto(values=np.asarray(data), shape=[1, len(data)], dtype=tf.float32)\n",
- " predict_response = mnist_predictor.predict(tensor_proto)\n",
+ " data = mnist.test.images[i]\n",
+ " if neo_optimize is False:\n",
+ " data = data.tolist()\n",
+ " data = tf.make_tensor_proto(values=np.asarray(data), shape=[1, len(data)], dtype=tf.float32)\n",
+ " predict_response = mnist_predictor.predict(data)\n",
  " \n",
  " print(\"========================================\")\n",
  " label = np.argmax(mnist.test.labels[i])\n",
  " print(\"label is {}\".format(label))\n",
- " prediction = predict_response['outputs']['classes']['int64_val'][0]\n",
+ " prediction = np.argmax(predict_response)\n",
+ " if neo_optimize is False:\n",
+ " prediction = predict_response['outputs']['classes']['int64_val'][0]\n",
  " print(\"prediction is {}\".format(prediction))"
  ]
  },
@@ -242,7 +292,6 @@
  }
  ],
  "metadata": {
- "notice": "Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.",
  "kernelspec": {
  "display_name": "Environment (conda_tensorflow_p27)",
  "language": "python",
@@ -259,7 +308,8 @@
  "nbconvert_exporter": "python",
  "pygments_lexer": "ipython3",
  "version": "2.7.13"
- }
+ },
+ "notice": "Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
  },
  "nbformat": 4,
  "nbformat_minor": 2

sagemaker_neo_compilation_jobs/README.md

@@ -0,0 +1,12 @@
+# Amazon SageMaker Examples
+
+### Amazon SageMaker Neo Compilation Jobs
+
+
+These examples focus on the Amazon SageMaker Neo which allows you to compile models and host in pre-built containers.
+
+- [Image Classification](imageclassification_caltech)
+- [MNIST with MXNet](mxnet_mnist)
+- [Deploying pre-trained PyTorch vision models](pytorch_torchvision)
+- [Distributed TensorFlow](tensorflow_distributed_mnist)
+- [Predicting Customer Churn](xgboost_customer_churn)

…ication-fulltraining-highlevel-neo.ipynb …ication-fulltraining-highlevel-neo.ipynbintroduction_to_amazon_algorithms/imageclassification_caltech/Image-classification-fulltraining-highlevel-neo.ipynb renamed to sagemaker_neo_compilation_jobs/imageclassification_caltech/Image-classification-fulltraining-highlevel-neo.ipynb introduction_to_amazon_algorithms/imageclassification_caltech/Image-classification-fulltraining-highlevel-neo.ipynb renamed to sagemaker_neo_compilation_jobs/imageclassification_caltech/Image-classification-fulltraining-highlevel-neo.ipynb

@@ -20,7 +20,8 @@
  "\n",
  "Welcome to our model optimization example for image classification.\n",
  "\n",
- "In this demo, we will use the Amazon sagemaker image classification algorithm to train on the [caltech-256 dataset](http://www.vision.caltech.edu/Image_Datasets/Caltech256/). \n",
+ "In this demo, we will use the Amazon sagemaker image classification algorithm to train on the [caltech-256 dataset](http://www.vision.caltech.edu/Image_Datasets/Caltech256/) and then we will demonstrate Amazon Sagemaker Neo's ability to optimize models.\n",
+ "\n",
  "\n",
  "To get started, we need to set up the environment with a few prerequisite steps, for permissions, configurations, and so on."
  ]
@@ -359,14 +360,25 @@
  "\n",
  "***\n",
  "\n",
- "Now we will test the trained model without any specific optimization for the hardware."
+ "We will use Sagemaker Neo to optimize the model."
  ]
  },
  {
  "cell_type": "markdown",
  "metadata": {},
  "source": [
- "## Optimize the model specifically for the architecture\n",
+ "## Introduction to SageMaker Neo\n",
+ "\n",
+ "***\n",
+ "\n",
+ "[Amazon SageMaker Neo](https://aws.amazon.com/sagemaker/neo/) optimizes models to run up to fourth as fast, with less than a tenth of the memory footprint, with no loss in accuracy. You start with a machine learning model built using MXNet, TensorFlow, PyTorch, or XGBoost and trained using Amazon SageMaker. Then you choose your target hardware platform from Intel, NVIDIA, or ARM. With a single click, SageMaker Neo will then compile the trained model into an executable. In this example, we will use the model we just trained and see how well the optimized model could perform."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Optimize the model specifically for the architecture using Neo API\n",
  "Now we will compare the same model, but compiled specifically for the architecture we're deploying on."
  ]
  },
@@ -391,10 +403,7 @@
  "metadata": {},
  "outputs": [],
  "source": [
- "# There is a known issue where SageMaker SDK locates the incorrect docker image URI for Image Classification\n",
- "# For now, we manually set Image URI\n",
  "optimized_ic.image = get_image_uri(sess.boto_region_name, 'image-classification-neo', repo_version=\"latest\")\n",
- "# There is a known issue where SageMaker SDK does not set the same. In the mean time we set the name\n",
  "optimized_ic.name = 'deployed-image-classification'"
  ]
  },
@@ -462,6 +471,16 @@
  "print(\"Result: label - \" + object_categories[index] + \", probability - \" + str(result[index]))"
  ]
  },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Conclusion\n",
+ "---\n",
+ "As you can notice the inference time using our neo-optimized model is better than the original one. SageMaker Neo automatically optimizes machine learning models to perform at up to fourth the speed with no loss in accuracy. In the diagram below shows you how our neo-optimized model performs 3x better with ResNet 152 in C5.9xlarge instance. The originl model stands for the uncompiled model deployed on Flask container on May 10th, 2019 and neo-optimized model stands for the compiled model deployed on Neo-AI-DLR container. The data for each trial is the average of 1000 trys for each endpoint.\n",
+ "![alt text](image-classification-latency.png \"Title\")\n"
+ ]
+ },
  {
  "cell_type": "markdown",
  "metadata": {},