TensorBoard TensorFlow example notebook (aws#1267) (aws#1269)

Co-authored-by: Yegor Tokmakov <yegor@tokmakov.biz>

Author: ngluna

aws_sagemaker_studio/frameworks/keras_pipe_mode_horovod/keras_pipe_mode_horovod_cifar10.ipynb

@@ -6,21 +6,28 @@
  "source": [
  "# Train and Host a Keras Model with Pipe Mode and Horovod on Amazon SageMaker\n",
  "\n",
- "*(This notebook was tested with the \"Python 3 (TensorFlow CPU Optimized)\" kernel.)*\n",
- "\n",
  "Amazon SageMaker is a fully-managed service that provides developers and data scientists with the ability to build, train, and deploy machine learning (ML) models quickly. Amazon SageMaker removes the heavy lifting from each step of the machine learning process to make it easier to develop high-quality models. The SageMaker Python SDK makes it easy to train and deploy models in Amazon SageMaker with several different machine learning and deep learning frameworks, including TensorFlow and Keras.\n",
  "\n",
  "In this notebook, we train and host a [Keras Sequential model](https://keras.io/getting-started/sequential-model-guide) on SageMaker. The model used for this notebook is a simple deep convolutional neural network (CNN) that was extracted from [the Keras examples](https://github.com/keras-team/keras/blob/master/examples/cifar10_cnn.py).\n",
  "\n",
- "For training our model, we also demonstrate distributed training with [Horovod](https://horovod.readthedocs.io) and Pipe Mode. Amazon SageMaker's Pipe Mode streams your dataset directly to your training instances instead of being downloaded first, which translates to training jobs that start sooner, finish quicker, and need less disk space."
+ "For training our model, we also demonstrate distributed training with [Horovod](https://horovod.readthedocs.io) and Pipe Mode. Amazon SageMaker's Pipe Mode streams your dataset directly to your training instances instead of being downloaded first, which translates to training jobs that start sooner, finish quicker, and need less disk space. \n",
+ "<br>\n",
+ "<br>\n",
+ "<b> Instance Type and Pricing: </b>\n",
+ "\n",
+ "This notebook was trained using the <b>Python 3 (TensorFlow CPU Optimized)</b> kernel using the <b>ml.p3.2xlarge</b> compute instance type in the <b>us-west-2 region</b>. Training time is approximately 70 minutes with the aforementioned hardware specifications.\n",
+ "\n",
+ "Price per hour depends on your region and instance type. You can reference prices on the [SageMaker pricing page](https://aws.amazon.com/sagemaker/pricing/). \n",
+ "\n",
+ "---\n",
+ "---"
  ]
  },
  {
  "cell_type": "markdown",
  "metadata": {},
  "source": [
  "## Setup\n",
- "\n",
  "First, we define a few variables that are be needed later in the example."
  ]
  },
@@ -161,7 +168,7 @@
  "from sagemaker.tensorflow import TensorFlow\n",
  "\n",
  "hyperparameters = {'epochs': 10, 'batch-size': 256}\n",
- "tags = [{'Key': 'Project', 'Value': 'cifar10'}, {'Key': 'TensorBoard', 'Value': 'file'}]\n",
+ "tags = [{'Key': 'Project', 'Value': 'cifar10'}]\n",
  "\n",
  "estimator = TensorFlow(entry_point='keras_cifar10.py',\n",
  " source_dir='source',\n",
@@ -171,7 +178,7 @@
  " framework_version='1.15.2',\n",
  " py_version='py3',\n",
  " train_instance_count=1,\n",
- " train_instance_type='ml.p2.xlarge',\n",
+ " train_instance_type='ml.p3.2xlarge',\n",
  " base_job_name='cifar10-tf',\n",
  " tags=tags)"
  ]
@@ -195,7 +202,7 @@
  " 'eval': '{}/eval'.format(dataset_uri),\n",
  "}\n",
  "\n",
- "estimator.fit(inputs, wait=True)"
+ "estimator.fit(inputs)"
  ]
  },
  {
@@ -273,26 +280,19 @@
  " framework_version='1.15.2',\n",
  " py_version='py3',\n",
  " train_instance_count=1,\n",
- " train_instance_type='ml.p2.xlarge',\n",
+ " train_instance_type='ml.p3.2xlarge',\n",
  " input_mode='Pipe',\n",
  " base_job_name='cifar10-tf-pipe',\n",
  " tags=tags)"
  ]
  },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Using the same training inputs from before, we call `fit()` on our estimator. Here, we set `wait=False`, but you can set `wait=True` if you would like to see the training logs in the notebook."
- ]
- },
  {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {},
  "outputs": [],
  "source": [
- "pipe_mode_estimator.fit(inputs, wait=False)"
+ "pipe_mode_estimator.fit(inputs)"
  ]
  },
  {
@@ -327,13 +327,6 @@
  "```python\n",
  "opt = Adam(lr=learning_rate * size, decay=weight_decay)\n",
  "opt = hvd.DistributedOptimizer(opt)\n",
- "```\n",
- "\n",
- "4. Choose to save checkpoints and send TensorBoard logs only from the master node:\n",
- "\n",
- "```python\n",
- "if hvd.rank() == 0:\n",
- " save_model(model, args.model_output_dir)\n",
  "```"
  ]
  },
@@ -380,60 +373,18 @@
  " framework_version='1.15.2',\n",
  " py_version='py3',\n",
  " train_instance_count=2,\n",
- " train_instance_type='ml.p2.xlarge',\n",
+ " train_instance_type='ml.p3.2xlarge',\n",
  " base_job_name='cifar10-tf-dist',\n",
  " tags=tags)"
  ]
  },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Like before, we call `fit()` on our estimator. If you want to see the training job logs in the notebook output, set `wait=True`."
- ]
- },
  {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {},
  "outputs": [],
  "source": [
- "dist_estimator.fit(inputs, wait=False)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Compare the training jobs with TensorBoard\n",
- "\n",
- "Using the visualization tool [TensorBoard](https://www.tensorflow.org/tensorboard), we can compare our training jobs.\n",
- "\n",
- "In a local setting, install TensorBoard with `pip install tensorboard`. Then run the command generated by the following code:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "!python generate_tensorboard_command.py"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "After running that command, we can access TensorBoard locally at http://localhost:6006.\n",
- "\n",
- "Based on the TensorBoard metrics, we can see that:\n",
- "1. All jobs run for 10 epochs (0 - 9).\n",
- "1. Both File Mode and Pipe Mode run for ~1 minute - Pipe Mode doesn't affect training performance.\n",
- "1. Distributed training runs for only 45 seconds.\n",
- "1. All of the training jobs resulted in similar validation accuracy.\n",
- "\n",
- "This example uses a relatively small dataset (179 MB). For larger datasets, Pipe Mode can significantly reduce training time because it does not copy the entire dataset into local memory."
+ "dist_estimator.fit(inputs)"
  ]
  },
  {
@@ -444,7 +395,11 @@
  "\n",
  "After we train our model, we can deploy it to a SageMaker Endpoint, which serves prediction requests in real-time. To do so, we simply call `deploy()` on our estimator, passing in the desired number of instances and instance type for the endpoint.\n",
  "\n",
- "Because we're using TensorFlow Serving for deployment, our training script saves the model in TensorFlow's SavedModel format. For more details, see [this blog post on deploying Keras and TF models in SageMaker](https://aws.amazon.com/blogs/machine-learning/deploy-trained-keras-or-tensorflow-models-using-amazon-sagemaker)."
+ "Because we're using TensorFlow Serving for deployment, our training script saves the model in TensorFlow's SavedModel format. \n",
+ "\n",
+ "We don't need accelerated computing power for inference, so let's switch over to a <b>ml.m4.xlarge</b> instance type. \n",
+ "\n",
+ "For more information about deploying Keras and TensorFlow models in SageMaker, see [this blog post](https://aws.amazon.com/blogs/machine-learning/deploy-trained-keras-or-tensorflow-models-using-amazon-sagemaker)."
  ]
  },
  {
@@ -604,7 +559,7 @@
  "kernelspec": {
  "display_name": "Python 3 (TensorFlow CPU Optimized)",
  "language": "python",
- "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-2:429704687514:image/tensorflow-1.15-cpu-py36"
+ "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-west-2:236514542706:image/tensorflow-1.15-cpu-py36"
  },
  "language_info": {
  "codemirror_mode": {
@@ -616,7 +571,7 @@
  "name": "python",
  "nbconvert_exporter": "python",
  "pygments_lexer": "ipython3",
- "version": "3.6.9"
+ "version": "3.6.3"
  },
  "notice": "Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.",
  "pycharm": {