Please provide the following information when requesting support.
• Hardware (T4/V100/Xavier/Nano/etc) T4
• Network Type (Detectnet_v2/Faster_rcnn/Yolo_v4/LPRnet/Mask_rcnn/Classification/etc) Detectnet_v2
• TLT Version (Please run “tlt info --verbose” and share “docker_tag” here)
• Training spec file(If have, please share here)detectnet_v2.ipynb
• How to reproduce the issue ? (This is for errors. Please share the command line and the detailed log here.)
When I used the detectnet_v2 algorithm to train my kitti format data set, I replaced the pretrained_resnet18 model with the dashcamnet model. My own data set categories are pedestrian and car. The following results appeared during training. There is only the accuracy of the pedestrian category, and the accuracy results of the other two categories are missing.
class name average precision (in %) ------------ -------------------------- car 0 cyclist 0 pedestrian 12.7891 Here are the detailed results.Please help me.
/usr/local/lib/python3.6/dist-packages/keras/engine/saving.py:292: UserWarning: No training configuration found in save file: the model was *not* compiled. Compile it manually. warnings.warn('No training configuration found in save file: ' _________________________________________________________ 2021-12-28 07:30:06,779 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Serial augmentation enabled = False 2021-12-28 07:30:06,779 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Pseudo sharding enabled = False 2021-12-28 07:30:06,779 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Max Image Dimensions (all sources): (0, 0) 2021-12-28 07:30:06,779 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: number of cpus: 48, io threads: 96, compute threads: 48, buffered batches: 4 2021-12-28 07:30:06,779 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: total dataset size 114, number of sources: 1, batch size per gpu: 4, steps: 29 WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead. 2021-12-28 07:30:06,831 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead. WARNING:tensorflow:Entity <bound method DriveNetTFRecordsParser.__call__ of <iva.detectnet_v2.dataloader.drivenet_dataloader.DriveNetTFRecordsParser object at 0x7f09cbe30fd0>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of <bound method DriveNetTFRecordsParser.__call__ of <iva.detectnet_v2.dataloader.drivenet_dataloader.DriveNetTFRecordsParser object at 0x7f09cbe30fd0>>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2021-12-28 07:30:06,893 [WARNING] tensorflow: Entity <bound method DriveNetTFRecordsParser.__call__ of <iva.detectnet_v2.dataloader.drivenet_dataloader.DriveNetTFRecordsParser object at 0x7f09cbe30fd0>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of <bound method DriveNetTFRecordsParser.__call__ of <iva.detectnet_v2.dataloader.drivenet_dataloader.DriveNetTFRecordsParser object at 0x7f09cbe30fd0>>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2021-12-28 07:30:06,922 [INFO] iva.detectnet_v2.dataloader.default_dataloader: Bounding box coordinates were detected in the input specification! Bboxes will be automatically converted to polygon coordinates. 2021-12-28 07:30:07,230 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: shuffle: True - shard 0 of 1 2021-12-28 07:30:07,238 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: sampling 1 datasets with weights: 2021-12-28 07:30:07,238 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: source: 0 weight: 1.000000 WARNING:tensorflow:Entity <bound method Processor.__call__ of <modulus.blocks.data_loaders.multi_source_loader.processors.asset_loader.AssetLoader object at 0x7f09b02cfc50>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of <bound method Processor.__call__ of <modulus.blocks.data_loaders.multi_source_loader.processors.asset_loader.AssetLoader object at 0x7f09b02cfc50>>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2021-12-28 07:30:07,259 [WARNING] tensorflow: Entity <bound method Processor.__call__ of <modulus.blocks.data_loaders.multi_source_loader.processors.asset_loader.AssetLoader object at 0x7f09b02cfc50>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of <bound method Processor.__call__ of <modulus.blocks.data_loaders.multi_source_loader.processors.asset_loader.AssetLoader object at 0x7f09b02cfc50>>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2021-12-28 07:30:07,692 [INFO] __main__: Found 114 samples in training set WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/rasterizers/bbox_rasterizer.py:347: The name tf.bincount is deprecated. Please use tf.math.bincount instead. 2021-12-28 07:30:07,832 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/rasterizers/bbox_rasterizer.py:347: The name tf.bincount is deprecated. Please use tf.math.bincount instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/training_proto_utilities.py:89: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead. 2021-12-28 07:30:07,981 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/training_proto_utilities.py:89: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/training_proto_utilities.py:36: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead. 2021-12-28 07:30:08,000 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/training_proto_utilities.py:36: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_functions.py:17: The name tf.log is deprecated. Please use tf.math.log instead. 2021-12-28 07:30:08,214 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_functions.py:17: The name tf.log is deprecated. Please use tf.math.log instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:235: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead. 2021-12-28 07:30:08,262 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:235: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/model/detectnet_model.py:587: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead. 2021-12-28 07:30:08,273 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/model/detectnet_model.py:587: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead. 2021-12-28 07:30:10,495 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Serial augmentation enabled = False 2021-12-28 07:30:10,495 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Pseudo sharding enabled = False 2021-12-28 07:30:10,495 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Max Image Dimensions (all sources): (0, 0) 2021-12-28 07:30:10,495 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: number of cpus: 48, io threads: 96, compute threads: 48, buffered batches: 4 2021-12-28 07:30:10,495 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: total dataset size 18, number of sources: 1, batch size per gpu: 4, steps: 5 WARNING:tensorflow:Entity <bound method DriveNetTFRecordsParser.__call__ of <iva.detectnet_v2.dataloader.drivenet_dataloader.DriveNetTFRecordsParser object at 0x7f09cbe30828>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of <bound method DriveNetTFRecordsParser.__call__ of <iva.detectnet_v2.dataloader.drivenet_dataloader.DriveNetTFRecordsParser object at 0x7f09cbe30828>>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2021-12-28 07:30:10,512 [WARNING] tensorflow: Entity <bound method DriveNetTFRecordsParser.__call__ of <iva.detectnet_v2.dataloader.drivenet_dataloader.DriveNetTFRecordsParser object at 0x7f09cbe30828>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of <bound method DriveNetTFRecordsParser.__call__ of <iva.detectnet_v2.dataloader.drivenet_dataloader.DriveNetTFRecordsParser object at 0x7f09cbe30828>>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2021-12-28 07:30:10,542 [INFO] iva.detectnet_v2.dataloader.default_dataloader: Bounding box coordinates were detected in the input specification! Bboxes will be automatically converted to polygon coordinates. 2021-12-28 07:30:10,823 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: shuffle: False - shard 0 of 1 2021-12-28 07:30:10,829 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: sampling 1 datasets with weights: 2021-12-28 07:30:10,829 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: source: 0 weight: 1.000000 WARNING:tensorflow:Entity <bound method Processor.__call__ of <modulus.blocks.data_loaders.multi_source_loader.processors.asset_loader.AssetLoader object at 0x7f09583e23c8>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of <bound method Processor.__call__ of <modulus.blocks.data_loaders.multi_source_loader.processors.asset_loader.AssetLoader object at 0x7f09583e23c8>>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2021-12-28 07:30:10,845 [WARNING] tensorflow: Entity <bound method Processor.__call__ of <modulus.blocks.data_loaders.multi_source_loader.processors.asset_loader.AssetLoader object at 0x7f09583e23c8>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of <bound method Processor.__call__ of <modulus.blocks.data_loaders.multi_source_loader.processors.asset_loader.AssetLoader object at 0x7f09583e23c8>>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2021-12-28 07:30:11,125 [INFO] __main__: Found 18 samples in validation set WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/validation_hook.py:40: The name tf.summary.FileWriterCache is deprecated. Please use tf.compat.v1.summary.FileWriterCache instead. 2021-12-28 07:30:11,861 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/validation_hook.py:40: The name tf.summary.FileWriterCache is deprecated. Please use tf.compat.v1.summary.FileWriterCache instead. 2021-12-28 07:30:13,292 [INFO] __main__: Checkpoint interval: 10 WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py:108: The name tf.train.Scaffold is deprecated. Please use tf.compat.v1.train.Scaffold instead. 2021-12-28 07:30:13,293 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py:108: The name tf.train.Scaffold is deprecated. Please use tf.compat.v1.train.Scaffold instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:14: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead. 2021-12-28 07:30:13,293 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:14: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:15: The name tf.tables_initializer is deprecated. Please use tf.compat.v1.tables_initializer instead. 2021-12-28 07:30:13,294 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:15: The name tf.tables_initializer is deprecated. Please use tf.compat.v1.tables_initializer instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:16: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead. 2021-12-28 07:30:13,295 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:16: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:59: The name tf.train.LoggingTensorHook is deprecated. Please use tf.estimator.LoggingTensorHook instead. 2021-12-28 07:30:13,298 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:59: The name tf.train.LoggingTensorHook is deprecated. Please use tf.estimator.LoggingTensorHook instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:60: The name tf.train.StopAtStepHook is deprecated. Please use tf.estimator.StopAtStepHook instead. 2021-12-28 07:30:13,298 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:60: The name tf.train.StopAtStepHook is deprecated. Please use tf.estimator.StopAtStepHook instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:73: The name tf.train.StepCounterHook is deprecated. Please use tf.estimator.StepCounterHook instead. 2021-12-28 07:30:13,298 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:73: The name tf.train.StepCounterHook is deprecated. Please use tf.estimator.StepCounterHook instead. INFO:tensorflow:Create CheckpointSaverHook. 2021-12-28 07:30:13,299 [INFO] tensorflow: Create CheckpointSaverHook. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:99: The name tf.train.SummarySaverHook is deprecated. Please use tf.estimator.SummarySaverHook instead. 2021-12-28 07:30:13,299 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:99: The name tf.train.SummarySaverHook is deprecated. Please use tf.estimator.SummarySaverHook instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:140: The name tf.train.SingularMonitoredSession is deprecated. Please use tf.compat.v1.train.SingularMonitoredSession instead. 2021-12-28 07:30:13,300 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:140: The name tf.train.SingularMonitoredSession is deprecated. Please use tf.compat.v1.train.SingularMonitoredSession instead. INFO:tensorflow:Graph was finalized. 2021-12-28 07:30:14,536 [INFO] tensorflow: Graph was finalized. INFO:tensorflow:Running local_init_op. 2021-12-28 07:30:16,823 [INFO] tensorflow: Running local_init_op. INFO:tensorflow:Done running local_init_op. 2021-12-28 07:30:17,613 [INFO] tensorflow: Done running local_init_op. INFO:tensorflow:Saving checkpoints for step-0. 2021-12-28 07:30:26,523 [INFO] tensorflow: Saving checkpoints for step-0. INFO:tensorflow:epoch = 0.0, learning_rate = 4.9999994e-06, loss = 0.09131467, step = 0 2021-12-28 07:30:57,713 [INFO] tensorflow: epoch = 0.0, learning_rate = 4.9999994e-06, loss = 0.09131467, step = 0 2021-12-28 07:30:57,722 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 0/120: loss: 0.09131 learning rate: 0.00000 Time taken: 0:00:00 ETA: 0:00:00 2021-12-28 07:30:57,722 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 0.439 INFO:tensorflow:global_step/sec: 0.772853 2021-12-28 07:31:00,306 [INFO] tensorflow: global_step/sec: 0.772853 INFO:tensorflow:global_step/sec: 10.915 2021-12-28 07:31:00,489 [INFO] tensorflow: global_step/sec: 10.915 INFO:tensorflow:global_step/sec: 10.8179 2021-12-28 07:31:00,674 [INFO] tensorflow: global_step/sec: 10.8179 INFO:tensorflow:global_step/sec: 10.6503 2021-12-28 07:31:00,862 [INFO] tensorflow: global_step/sec: 10.6503 INFO:tensorflow:global_step/sec: 11.1249 2021-12-28 07:31:01,042 [INFO] tensorflow: global_step/sec: 11.1249 INFO:tensorflow:global_step/sec: 11.0209 2021-12-28 07:31:01,223 [INFO] tensorflow: global_step/sec: 11.0209 INFO:tensorflow:global_step/sec: 10.5862 2021-12-28 07:31:01,412 [INFO] tensorflow: global_step/sec: 10.5862 INFO:tensorflow:global_step/sec: 10.0341 2021-12-28 07:31:01,611 [INFO] tensorflow: global_step/sec: 10.0341 INFO:tensorflow:global_step/sec: 11.1387 2021-12-28 07:31:01,791 [INFO] tensorflow: global_step/sec: 11.1387 INFO:tensorflow:global_step/sec: 9.6168 2021-12-28 07:31:01,999 [INFO] tensorflow: global_step/sec: 9.6168 INFO:tensorflow:global_step/sec: 9.13294 2021-12-28 07:31:02,218 [INFO] tensorflow: global_step/sec: 9.13294 INFO:tensorflow:global_step/sec: 10.1474 2021-12-28 07:31:02,415 [INFO] tensorflow: global_step/sec: 10.1474 2021-12-28 07:31:02,416 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 7.246 INFO:tensorflow:global_step/sec: 10.1296 2021-12-28 07:31:02,612 [INFO] tensorflow: global_step/sec: 10.1296 INFO:tensorflow:epoch = 0.9655172413793103, learning_rate = 7.242517e-06, loss = 0.06743895, step = 28 (5.111 sec) 2021-12-28 07:31:02,824 [INFO] tensorflow: epoch = 0.9655172413793103, learning_rate = 7.242517e-06, loss = 0.06743895, step = 28 (5.111 sec) INFO:tensorflow:global_step/sec: 9.23476 2021-12-28 07:31:02,829 [INFO] tensorflow: global_step/sec: 9.23476 49da551de758:116:224 [0] NCCL INFO Bootstrap : Using [0]lo:127.0.0.1<0> [1]eth0:172.17.0.7<0> 49da551de758:116:224 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation 49da551de758:116:224 [0] NCCL INFO NET/IB : No device found. 49da551de758:116:224 [0] NCCL INFO NET/Socket : Using [0]lo:127.0.0.1<0> [1]eth0:172.17.0.7<0> 49da551de758:116:224 [0] NCCL INFO Using network Socket NCCL version 2.7.8+cuda11.1 49da551de758:116:224 [0] NCCL INFO Channel 00/32 : 0 49da551de758:116:224 [0] NCCL INFO Channel 01/32 : 0 49da551de758:116:224 [0] NCCL INFO Channel 02/32 : 0 .... Median Inference Time: 0.008003 INFO:tensorflow:epoch = 110.0, learning_rate = 1.7969065e-05, loss = 8.230345e-05, step = 3190 (9.965 sec) 2021-12-28 07:37:44,296 [INFO] tensorflow: epoch = 110.0, learning_rate = 1.7969065e-05, loss = 8.230345e-05, step = 3190 (9.965 sec) INFO:tensorflow:global_step/sec: 0.216576 2021-12-28 07:37:44,298 [INFO] tensorflow: global_step/sec: 0.216576 2021-12-28 07:37:44,302 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 110/120: loss: 0.00008 learning rate: 0.00002 Time taken: 0:00:11.694104 ETA: 0:01:56.941044 INFO:tensorflow:global_step/sec: 10.2147 .... INFO:tensorflow:global_step/sec: 10.9742 2021-12-28 07:38:07,158 [INFO] tensorflow: global_step/sec: 10.9742 INFO:tensorflow:global_step/sec: 11.0977 2021-12-28 07:38:07,339 [INFO] tensorflow: global_step/sec: 11.0977 INFO:tensorflow:global_step/sec: 10.3461 2021-12-28 07:38:07,532 [INFO] tensorflow: global_step/sec: 10.3461 INFO:tensorflow:global_step/sec: 11.9319 2021-12-28 07:38:07,700 [INFO] tensorflow: global_step/sec: 11.9319 INFO:tensorflow:global_step/sec: 10.8791 2021-12-28 07:38:07,883 [INFO] tensorflow: global_step/sec: 10.8791 2021-12-28 07:38:07,975 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 43.965 INFO:tensorflow:global_step/sec: 10.9462 2021-12-28 07:38:08,066 [INFO] tensorflow: global_step/sec: 10.9462 2021-12-28 07:38:08,166 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 119/120: loss: 0.00007 learning rate: 0.00001 Time taken: 0:00:02.648905 ETA: 0:00:02.648905 INFO:tensorflow:global_step/sec: 10.6033 2021-12-28 07:38:08,255 [INFO] tensorflow: global_step/sec: 10.6033 INFO:tensorflow:global_step/sec: 10.9142 2021-12-28 07:38:08,438 [INFO] tensorflow: global_step/sec: 10.9142 INFO:tensorflow:global_step/sec: 10.5721 2021-12-28 07:38:08,627 [INFO] tensorflow: global_step/sec: 10.5721 INFO:tensorflow:global_step/sec: 11.1025 2021-12-28 07:38:08,807 [INFO] tensorflow: global_step/sec: 11.1025 INFO:tensorflow:global_step/sec: 11.317 2021-12-28 07:38:08,984 [INFO] tensorflow: global_step/sec: 11.317 INFO:tensorflow:global_step/sec: 10.2896 2021-12-28 07:38:09,178 [INFO] tensorflow: global_step/sec: 10.2896 INFO:tensorflow:global_step/sec: 10.7394 2021-12-28 07:38:09,365 [INFO] tensorflow: global_step/sec: 10.7394 INFO:tensorflow:global_step/sec: 11.306 2021-12-28 07:38:09,542 [INFO] tensorflow: global_step/sec: 11.306 INFO:tensorflow:global_step/sec: 10.4033 2021-12-28 07:38:09,734 [INFO] tensorflow: global_step/sec: 10.4033 INFO:tensorflow:epoch = 119.6551724137931, learning_rate = 5.225487e-06, loss = 0.00011281592, step = 3470 (5.146 sec) 2021-12-28 07:38:09,919 [INFO] tensorflow: epoch = 119.6551724137931, learning_rate = 5.225487e-06, loss = 0.00011281592, step = 3470 (5.146 sec) INFO:tensorflow:global_step/sec: 10.5473 2021-12-28 07:38:09,924 [INFO] tensorflow: global_step/sec: 10.5473 INFO:tensorflow:global_step/sec: 11.345 2021-12-28 07:38:10,100 [INFO] tensorflow: global_step/sec: 11.345 INFO:tensorflow:global_step/sec: 11.0311 2021-12-28 07:38:10,281 [INFO] tensorflow: global_step/sec: 11.0311 2021-12-28 07:38:10,284 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 43.326 INFO:tensorflow:global_step/sec: 11.2454 2021-12-28 07:38:10,459 [INFO] tensorflow: global_step/sec: 11.2454 INFO:tensorflow:global_step/sec: 11.3238 2021-12-28 07:38:10,636 [INFO] tensorflow: global_step/sec: 11.3238 INFO:tensorflow:Saving checkpoints for step-3480. 2021-12-28 07:38:10,736 [INFO] tensorflow: Saving checkpoints for step-3480. WARNING:tensorflow:Ignoring: /tmp/tmpi08w9yct; No such file or directory 2021-12-28 07:38:10,979 [WARNING] tensorflow: Ignoring: /tmp/tmpi08w9yct; No such file or directory 2021-12-28 07:38:15,057 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 4, 0.00s/step Matching predictions to ground truth, class 1/3.: 100%|█| 3282/3282 [00:00<00:00, 24675.21it/s] Matching predictions to ground truth, class 3/3.: 100%|█| 1341/1341 [00:00<00:00, 15211.67it/s] Epoch 120/120 ========================= Validation cost: 0.000036 Mean average_precision (in %): 4.2630 class name average precision (in %) ------------ -------------------------- car 0 cyclist 0 pedestrian 12.7891 Median Inference Time: 0.007898 WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead. 2021-12-28 07:38:20,386 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:98: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead. 2021-12-28 07:38:20,386 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:98: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead. 2021-12-28 07:38:20,390 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 43.326 Time taken to run __main__:main: 0:08:27.855154. 2021-12-28 15:38:26,442 [INFO] tlt.components.docker_handler.docker_handler: Stopping container. 
