Hello, I am running into an issue similar to this one when trying to run evaluation with an unpruned, un-trained, TrafficCamNet from detectnet_v2 with TAO toolkit version 4.0.1.
Command:
%%bash -s "$tao_container_id" "$TRAFFIC_CAM_MODEL_SPEC_PATH" "$TRAFFIC_CAM_MODEL_PATH" "$KEY" docker exec $1 detectnet_v2 evaluate --experiment_spec_file $2 --model_file $3 --key $4 --framework tlt produces the following exception:
2023-07-28 15:40:44.038041: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0 Using TensorFlow backend. WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them. WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them. /usr/local/lib/python3.6/dist-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (3.0.4) doesn't match a supported version! RequestsDependencyWarning) Using TensorFlow backend. 2023-07-28 15:40:52,430 [INFO] iva.detectnet_v2.spec_handler.spec_loader: Merging specification from /workspace/tao-eval/specs/traffic_cam_model_spec.txt WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:153: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead. 2023-07-28 15:40:52,435 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:153: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead. 2023-07-28 15:40:53,235 [INFO] root: Loading model weights. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead. 2023-07-28 15:40:53,678 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead. 2023-07-28 15:40:53,689 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead. 2023-07-28 15:40:53,721 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead. 2023-07-28 15:40:54,945 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:190: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead. 2023-07-28 15:40:54,945 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:190: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead. 2023-07-28 15:40:54,946 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead. 2023-07-28 15:40:55,298 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead. /usr/local/lib/python3.6/dist-packages/keras/engine/saving.py:292: UserWarning: No training configuration found in save file: the model was *not* compiled. Compile it manually. warnings.warn('No training configuration found in save file: ' 2023-07-28 15:40:56,006 [INFO] iva.detectnet_v2.objectives.bbox_objective: Default L1 loss function will be used. 2023-07-28 15:40:56,006 [INFO] root: Building dataloader. 2023-07-28 15:40:57,331 [INFO] root: Sampling mode of the dataloader was set to user_defined. 2023-07-28 15:40:57,332 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Serial augmentation enabled = False 2023-07-28 15:40:57,332 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Pseudo sharding enabled = False 2023-07-28 15:40:57,332 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Max Image Dimensions (all sources): (0, 0) 2023-07-28 15:40:57,332 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: number of cpus: 24, io threads: 48, compute threads: 24, buffered batches: 4 2023-07-28 15:40:57,333 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: total dataset size 10552, number of sources: 1, batch size per gpu: 4, steps: 2638 WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead. 2023-07-28 15:40:57,373 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead. WARNING:tensorflow:Entity <bound method DriveNetTFRecordsParser.__call__ of <iva.detectnet_v2.dataloader.drivenet_dataloader.DriveNetTFRecordsParser object at 0x7ff743a1fe10>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of <bound method DriveNetTFRecordsParser.__call__ of <iva.detectnet_v2.dataloader.drivenet_dataloader.DriveNetTFRecordsParser object at 0x7ff743a1fe10>>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2023-07-28 15:40:57,415 [WARNING] tensorflow: Entity <bound method DriveNetTFRecordsParser.__call__ of <iva.detectnet_v2.dataloader.drivenet_dataloader.DriveNetTFRecordsParser object at 0x7ff743a1fe10>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of <bound method DriveNetTFRecordsParser.__call__ of <iva.detectnet_v2.dataloader.drivenet_dataloader.DriveNetTFRecordsParser object at 0x7ff743a1fe10>>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2023-07-28 15:40:57,439 [INFO] iva.detectnet_v2.dataloader.default_dataloader: Bounding box coordinates were detected in the input specification! Bboxes will be automatically converted to polygon coordinates. 2023-07-28 15:40:57,747 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: shuffle: False - shard 0 of 1 2023-07-28 15:40:57,755 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: sampling 1 datasets with weights: 2023-07-28 15:40:57,755 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: source: 0 weight: 1.000000 WARNING:tensorflow:Entity <bound method Processor.__call__ of <modulus.blocks.data_loaders.multi_source_loader.processors.asset_loader.AssetLoader object at 0x7ff71fff6dd8>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of <bound method Processor.__call__ of <modulus.blocks.data_loaders.multi_source_loader.processors.asset_loader.AssetLoader object at 0x7ff71fff6dd8>>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2023-07-28 15:40:57,774 [WARNING] tensorflow: Entity <bound method Processor.__call__ of <modulus.blocks.data_loaders.multi_source_loader.processors.asset_loader.AssetLoader object at 0x7ff71fff6dd8>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of <bound method Processor.__call__ of <modulus.blocks.data_loaders.multi_source_loader.processors.asset_loader.AssetLoader object at 0x7ff71fff6dd8>>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2023-07-28 15:40:58,081 [INFO] iva.detectnet_v2.evaluation.build_evaluator: Found 10552 samples in validation set Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 1607, in _create_c_op c_op = c_api.TF_FinishOperation(op_desc) tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot reshape a tensor with 130560 elements to shape [4,1,4,34,60] (32640 elements) for 'reshape_1_1/Reshape' (op: 'Reshape') with input shapes: [4,16,34,60], [5] and with input tensors computed as partial shapes: input[1] = [4,1,4,34,60]. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "</usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/scripts/evaluate.py>", line 3, in <module> File "<frozen iva.detectnet_v2.scripts.evaluate>", line 204, in <module> File "<frozen iva.detectnet_v2.scripts.evaluate>", line 194, in <module> File "<decorator-gen-2>", line 2, in main File "<frozen iva.detectnet_v2.utilities.timer>", line 46, in wrapped_fn File "<frozen iva.detectnet_v2.scripts.evaluate>", line 177, in main File "<frozen iva.detectnet_v2.evaluation.build_evaluator>", line 158, in build_evaluator_for_trained_gridbox File "<frozen iva.detectnet_v2.model.utilities>", line 30, in _fn_wrapper File "<frozen iva.detectnet_v2.model.detectnet_model>", line 736, in build_validation_graph File "<frozen iva.detectnet_v2.model.utilities>", line 30, in _fn_wrapper File "<frozen iva.detectnet_v2.model.detectnet_model>", line 690, in build_inference_graph File "<frozen iva.detectnet_v2.model.detectnet_model>", line 306, in predictions_to_dict File "<frozen iva.detectnet_v2.objectives.base_objective>", line 98, in reshape_output File "/usr/local/lib/python3.6/dist-packages/keras/engine/base_layer.py", line 457, in __call__ output = self.call(inputs, **kwargs) File "/usr/local/lib/python3.6/dist-packages/keras/layers/core.py", line 401, in call return K.reshape(inputs, (K.shape(inputs)[0],) + self.target_shape) File "/usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py", line 1969, in reshape return tf.reshape(x, shape) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/array_ops.py", line 131, in reshape result = gen_array_ops.reshape(tensor, shape, name) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gen_array_ops.py", line 8115, in reshape "Reshape", tensor=tensor, shape=shape, name=name) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper op_def=op_def) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/deprecation.py", line 513, in new_func return func(*args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op attrs, op_def, compute_device) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal op_def=op_def) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 1770, in __init__ control_input_ops) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 1610, in _create_c_op raise ValueError(str(e)) ValueError: Cannot reshape a tensor with 130560 elements to shape [4,1,4,34,60] (32640 elements) for 'reshape_1_1/Reshape' (op: 'Reshape') with input shapes: [4,16,34,60], [5] and with input tensors computed as partial shapes: input[1] = [4,1,4,34,60]. /usr/local/lib/python3.6/dist-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (3.0.4) doesn't match a supported version! RequestsDependencyWarning) Telemetry data couldn't be sent, but the command ran successfully. [WARNING]: __init__() missing 4 required positional arguments: 'code', 'msg', 'hdrs', and 'fp' Execution status: FAIL Please provide the following information when requesting support.
• Hardware: Tesla T4
• Network Type: Detectnet_v2/TrafficCamNet) (unpruned)
• Training spec file (see below)
random_seed: 23 model_config { arch: "resnet" pretrained_model_file: "/workspace/tao-eval/trafficcamnet/unpruned_v1.0/trafficcamnet_vunpruned_v1.0/resnet18_trafficcamnet.tlt" num_layers: 18 use_batch_norm: true objective_set { bbox { scale: 35.0 offset: 0.5 } cov { } } training_precision { backend_floatx: FLOAT32 } all_projections: true } evaluation_config { validation_period_during_training: 10 first_validation_epoch: 30 average_precision_mode: INTEGRATE minimum_detection_ground_truth_overlap { key: "car" value: 0.6 } evaluation_box_config { key: "car" value { minimum_height: 20 maximum_height: 9999 minimum_width: 10 maximum_width: 9999 } } } dataset_config { data_sources { tfrecords_path: "/workspace/tao-eval/data/tfrecords/*" image_directory_path: "/workspace/tao-eval/coco/data" } image_extension: "jpg" target_class_mapping { key: "car" value: "car" } validation_fold: 0 } augmentation_config { preprocessing { output_image_width: 960 output_image_height: 544 min_bbox_width: 1.0 min_bbox_height: 1.0 output_image_channel: 3 enable_auto_resize: true } spatial_augmentation { hflip_probability: 0.5 zoom_min: 1.0 zoom_max: 1.0 translate_max_x: 8.0 translate_max_y: 8.0 } color_augmentation { hue_rotation_max: 25.0 saturation_shift_max: 0.2 contrast_scale_max: 0.1 contrast_center: 0.5 } } # Taken from notebook example: postprocessing_config { target_class_config { key: "car" value { clustering_config { clustering_algorithm: DBSCAN dbscan_confidence_threshold: 0.9 coverage_threshold: 0.0 dbscan_eps: 0.2 dbscan_min_samples: 0.05 minimum_bounding_box_height: 20 } } } } # Taken from notebook example: cost_function_config { target_classes { name: "car" class_weight: 1.0 coverage_foreground_weight: 0.05 objectives { name: "cov" initial_weight: 1.0 weight_target: 1.0 } objectives { name: "bbox" initial_weight: 10.0 weight_target: 10.0 } } } # Taken from notebook example: training_config { batch_size_per_gpu: 4 num_epochs: 120 learning_rate { soft_start_annealing_schedule { min_learning_rate: 5e-06 max_learning_rate: 5e-04 soft_start: 0.1 annealing: 0.7 } } regularizer { type: L1 weight: 3e-09 } optimizer { adam { epsilon: 10e-09 beta1: 0.9 beta2: 1.0 } } cost_scaling { initial_exponent: 20.0 increment: 0.005 decrement: 1.0 } visualizer { enabled: true num_images: 3 scalar_logging_frequency: 50 infrequent_logging_frequency: 5 target_class_config { key: "car" value: { coverage_threshold: 0.005 } } } checkpoint_interval: 10 } bbox_rasterizer_config { target_class_config { key: "car" value { cov_center_x: 0.5 cov_center_y: 0.5 cov_radius_x: 1.0 cov_radius_y: 1.0 bbox_min_radius: 1.0 } } deadzone_radius: 0.4 } My coco_config file is used for dataset conversion:
coco_traffic_cam_spec.txt:
coco_config { root_directory_path: "/workspace/tao-eval/coco/" img_dir_names: ["data"] annotation_files: ["labels.json"] num_partitions: 1 num_shards: [8] } image_directory_path: "/workspace/tao-eval/coco/" target_class_mapping { key: "vehicle" # Label from dataset value: "car" # Label in TrafficCamNet } which works properly for the following command:
%%bash -s "$tao_container_id" "$TRAFFIC_CAM_DATA_SPEC_PATH" "$CONVERT_OUTPUT_DIR" docker exec $1 detectnet_v2 dataset_convert --dataset_export_spec $2 --output_filename $3 with output:
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them. /usr/local/lib/python3.6/dist-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (3.0.4) doesn't match a supported version! RequestsDependencyWarning) Using TensorFlow backend. 2023-07-28 15:40:11,559 [INFO] iva.detectnet_v2.dataio.build_converter: Instantiating a coco converter 2023-07-28 15:40:12,287 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 0, shard 0 2023-07-28 15:40:12,613 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 0, shard 1 2023-07-28 15:40:12,951 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 0, shard 2 2023-07-28 15:40:13,293 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 0, shard 3 2023-07-28 15:40:13,612 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 0, shard 4 2023-07-28 15:40:13,925 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 0, shard 5 2023-07-28 15:40:14,214 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 0, shard 6 2023-07-28 15:40:14,489 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 0, shard 7 2023-07-28 15:40:14,765 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Wrote the following numbers of objects: b'vehicle': 38956 2023-07-28 15:40:14,765 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 1, shard 0 2023-07-28 15:40:15,097 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 1, shard 1 2023-07-28 15:40:15,439 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 1, shard 2 2023-07-28 15:40:15,758 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 1, shard 3 2023-07-28 15:40:16,084 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 1, shard 4 2023-07-28 15:40:16,406 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 1, shard 5 2023-07-28 15:40:16,695 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 1, shard 6 2023-07-28 15:40:16,983 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Writing partition 1, shard 7 2023-07-28 15:40:17,276 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Wrote the following numbers of objects: b'vehicle': 38956 2023-07-28 15:40:17,276 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Cumulative object statistics 2023-07-28 15:40:17,276 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Wrote the following numbers of objects: b'vehicle': 77912 2023-07-28 15:40:17,276 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Class map. Label in GT: Label in tfrecords file vehicle: vehicle 2023-07-28 15:40:17,276 [INFO] iva.detectnet_v2.dataio.coco_converter_lib: Tfrecords generation complete. loading annotations into memory... Done (t=0.26s) creating index... index created! loading annotations into memory... Done (t=0.42s) creating index... index created! For the dataset_config in the experiment_spec, please use labels in the tfrecords file, while writing the classmap. Note that in my training spec I have preprocessing in augmentation_config set to the output dimensions expected by TrafficCamNet, which is 960x544.
My TFRecord input files are of uniform dimension (640x480) with a single label class. Dimension uniformity confirmed by this script:
import tensorflow as tf import os tfrecord_dir = "/workspace/tao-eval/data/tfrecords/" files = os.listdir(tfrecord_dir) expected_frame_width = 640 expected_frame_height = 480 found_width = [] found_height = [] for file in files: full_path = tfrecord_dir + file tfrecords_iter = tf.python_io.tf_record_iterator(full_path) for record in tfrecords_iter: example = tf.train.Example() example.ParseFromString(record) features = example.features.feature frame_width = features["frame/width"].int64_list.value frame_height = features["frame/height"].int64_list.value if frame_width[0] not in found_width: found_width.append(frame_width[0]) if frame_height[0] not in found_height: found_height.append(frame_height[0]) if frame_width[0] != expected_frame_width: print("The frame_id is: {}\n".format(frame_id)) print("The frame_width is:{}\n".format(frame_width)) print("The frame_height:{}\n".format(frame_height)) elif frame_height[0] != expected_frame_height: print("The frame_id is: {}\n".format(frame_id)) print("The frame_width is:{}\n".format(frame_width)) print("The frame_height:{}\n".format(frame_height)) print(found_width) print(found_height) I expect the resizing from (640x480) to the needed (960x544) to be taken care of by the enable_auto_resize: true in the augmentation_config.preprocessing block.
So my guess is my issue is related to the fact that my source dataset has 5 classes (the only one I care about is vehicle, so I only map ‘vehicle’ to ‘car’ in target_class_mapping, but TrafficCamNet was trained on 4 classes. Are there best practices for dealing with unused classes from the source dataset? I have also tried evaluation with a subset of my training data that consists of only “vehicle” detections with the same issue.
Last thing: it seems that you usually run evaluate after train, but I am running evaluate first in order to establish a performance baseline before fine-tuning.
Any advice appreciated. Thank you.