- Notifications
You must be signed in to change notification settings - Fork 45.4k
Description
Prerequisites
Please answer the following questions for yourself before submitting an issue.
- I am using the latest TensorFlow Model Garden release and TensorFlow 2.
- I am reporting the issue to the correct repository. (Model Garden official or research directory)
- I checked to make sure that this issue has not already been filed.
1. The entire URL of the file you are using
https://github.com/tensorflow/models/blob/master/research/object_detection/exporter_main_v2.py
2. Describe the bug
I'm trying to export a TF2 OD API efficientdet_d0 that I just trained (using pretrained weights from model zoo 2 and 2GPUs in the tf2 docker image from OD API repo) with exporter_main_v2.py in order to serve this model with Tensorflow Serving.
When exporting, I'm getting a type error.
3. Steps to reproduce
Take the result of training a ssd_efficientdet_d0_512x512 model.
Inside TF2 container, (https://github.com/tensorflow/models/blob/master/research/object_detection/dockerfiles/tf2/Dockerfile), run :
python object_detection/exporter_main_v2.py \ --input_type encoded_image_string_tensor \ --trained_checkpoint_dir /home/tensorflow/outputs/efficientdet_d0_10k_16b_512_2gpu/model_dir/ \ --output_directory /home/tensorflow/outputs/efficientdet_d0_10k_16b_512_2gpu/export/ \ --pipeline_config_path /home/tensorflow/pipeline.config4. Expected behavior
The model is exported in /home/tensorflow/outputs/efficientdet_d0_10k_16b_512_2gpu/export/ as described in https://github.com/tensorflow/models/blob/master/research/object_detection/exporter_main_v2.py
5. Additional context
Logs :
I0711 17:37:09.737181 139828628232000 ssd_efficientnet_bifpn_feature_extractor.py:144] EfficientDet EfficientNet backbone version: efficientnet-b0 I0711 17:37:09.737298 139828628232000 ssd_efficientnet_bifpn_feature_extractor.py:145] EfficientDet BiFPN num filters: 64 I0711 17:37:09.737352 139828628232000 ssd_efficientnet_bifpn_feature_extractor.py:147] EfficientDet BiFPN num iterations: 3 I0711 17:37:09.745369 139828628232000 efficientnet_model.py:146] round_filter input=32 output=32 I0711 17:37:10.747392 139828628232000 efficientnet_model.py:146] round_filter input=32 output=32 I0711 17:37:10.747502 139828628232000 efficientnet_model.py:146] round_filter input=16 output=16 I0711 17:37:10.829591 139828628232000 efficientnet_model.py:146] round_filter input=16 output=16 I0711 17:37:10.829701 139828628232000 efficientnet_model.py:146] round_filter input=24 output=24 I0711 17:37:11.092894 139828628232000 efficientnet_model.py:146] round_filter input=24 output=24 I0711 17:37:11.093004 139828628232000 efficientnet_model.py:146] round_filter input=40 output=40 I0711 17:37:11.314886 139828628232000 efficientnet_model.py:146] round_filter input=40 output=40 I0711 17:37:11.314993 139828628232000 efficientnet_model.py:146] round_filter input=80 output=80 I0711 17:37:11.654193 139828628232000 efficientnet_model.py:146] round_filter input=80 output=80 I0711 17:37:11.654304 139828628232000 efficientnet_model.py:146] round_filter input=112 output=112 I0711 17:37:11.993307 139828628232000 efficientnet_model.py:146] round_filter input=112 output=112 I0711 17:37:11.993419 139828628232000 efficientnet_model.py:146] round_filter input=192 output=192 I0711 17:37:12.449521 139828628232000 efficientnet_model.py:146] round_filter input=192 output=192 I0711 17:37:12.449632 139828628232000 efficientnet_model.py:146] round_filter input=320 output=320 I0711 17:37:12.555558 139828628232000 efficientnet_model.py:146] round_filter input=1280 output=1280 I0711 17:37:12.599392 139828628232000 efficientnet_model.py:459] Building model efficientnet with params ModelConfig(width_coefficient=1.0, depth_coefficient=1.0, resolution=224, dropout_rate=0.2, blocks=(BlockConfig(input_filters=32, output_filters=16, kernel_size=3, num_repeat=1, expand_ratio=1, strides=(1, 1), se_ratio=0.25, id_skip=True, fused_conv=False, conv_type='depthwise'), BlockConfig(input_filters=16, output_filters=24, kernel_size=3, num_repeat=2, expand_ratio=6, strides=(2, 2), se_ratio=0.25, id_skip=True, fused_conv=False, conv_type='depthwise'), BlockConfig(input_filters=24, output_filters=40, kernel_size=5, num_repeat=2, expand_ratio=6, strides=(2, 2), se_ratio=0.25, id_skip=True, fused_conv=False, conv_type='depthwise'), BlockConfig(input_filters=40, output_filters=80, kernel_size=3, num_repeat=3, expand_ratio=6, strides=(2, 2), se_ratio=0.25, id_skip=True, fused_conv=False, conv_type='depthwise'), BlockConfig(input_filters=80, output_filters=112, kernel_size=5, num_repeat=3, expand_ratio=6, strides=(1, 1), se_ratio=0.25, id_skip=True, fused_conv=False, conv_type='depthwise'), BlockConfig(input_filters=112, output_filters=192, kernel_size=5, num_repeat=4, expand_ratio=6, strides=(2, 2), se_ratio=0.25, id_skip=True, fused_conv=False, conv_type='depthwise'), BlockConfig(input_filters=192, output_filters=320, kernel_size=3, num_repeat=1, expand_ratio=6, strides=(1, 1), se_ratio=0.25, id_skip=True, fused_conv=False, conv_type='depthwise')), stem_base_filters=32, top_base_filters=1280, activation='simple_swish', batch_norm='default', bn_momentum=0.99, bn_epsilon=0.001, weight_decay=5e-06, drop_connect_rate=0.2, depth_divisor=8, min_depth=None, use_se=True, input_channels=3, num_classes=1000, model_name='efficientnet', rescale_input=False, data_format='channels_last', dtype='float32') WARNING:tensorflow:From /home/tensorflow/.local/lib/python3.6/site-packages/object_detection/exporter_lib_v2.py:101: calling map_fn_v2 (from tensorflow.python.ops.map_fn) with back_prop=False is deprecated and will be removed in a future version. Instructions for updating: back_prop=False is deprecated. Consider using tf.stop_gradient instead. Instead of: results = tf.map_fn(fn, elems, back_prop=False) Use: results = tf.nest.map_structure(tf.stop_gradient, tf.map_fn(fn, elems)) W0711 17:37:16.879535 139828628232000 deprecation.py:573] From /home/tensorflow/.local/lib/python3.6/site-packages/object_detection/exporter_lib_v2.py:101: calling map_fn_v2 (from tensorflow.python.ops.map_fn) with back_prop=False is deprecated and will be removed in a future version. Instructions for updating: back_prop=False is deprecated. Consider using tf.stop_gradient instead. Instead of: results = tf.map_fn(fn, elems, back_prop=False) Use: results = tf.nest.map_structure(tf.stop_gradient, tf.map_fn(fn, elems)) WARNING:tensorflow:Skipping full serialization of Keras layer <object_detection.meta_architectures.ssd_meta_arch.SSDMetaArch object at 0x7f2b7c1ac5f8>, because it is not built. W0711 17:37:39.926661 139828628232000 save_impl.py:76] Skipping full serialization of Keras layer <object_detection.meta_architectures.ssd_meta_arch.SSDMetaArch object at 0x7f2b7c1ac5f8>, because it is not built. Traceback (most recent call last): File "object_detection/exporter_main_v2.py", line 126, in <module> app.run(main) File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 299, in run _run_main(main, args) File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 250, in _run_main sys.exit(main(argv)) File "object_detection/exporter_main_v2.py", line 122, in main FLAGS.output_directory) File "/home/tensorflow/.local/lib/python3.6/site-packages/object_detection/exporter_lib_v2.py", line 180, in export_inference_graph signatures=concrete_function) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/saved_model/save.py", line 951, in save obj, export_dir, signatures, options, meta_graph_def) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/saved_model/save.py", line 1022, in _build_meta_graph _ = _SaveableView(checkpoint_graph_view) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/saved_model/save.py", line 173, in __init__ self.checkpoint_view.objects_ids_and_slot_variables()) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/training/tracking/graph_view.py", line 415, in objects_ids_and_slot_variables trackable_objects, path_to_root = self._breadth_first_traversal() File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/training/tracking/graph_view.py", line 199, in _breadth_first_traversal for name, dependency in self.list_dependencies(current_trackable): File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/saved_model/save.py", line 108, in list_dependencies extra_dependencies = self.list_extra_dependencies(obj) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/saved_model/save.py", line 137, in list_extra_dependencies self._serialization_cache) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 2746, in _list_extra_dependencies_for_serialization .list_extra_dependencies_for_serialization(serialization_cache)) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/base_serialization.py", line 74, in list_extra_dependencies_for_serialization return self.objects_to_serialize(serialization_cache) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/layer_serialization.py", line 73, in objects_to_serialize serialization_cache).objects_to_serialize) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/layer_serialization.py", line 92, in _get_serialized_attributes serialization_cache) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/model_serialization.py", line 53, in _get_serialized_attributes_internal serialization_cache)) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/layer_serialization.py", line 101, in _get_serialized_attributes_internal functions = save_impl.wrap_layer_functions(self.obj, serialization_cache) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 163, in wrap_layer_functions '{}_layer_call_and_return_conditional_losses'.format(layer.name)) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 503, in add_function self.add_trace(*self._input_signature) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 418, in add_trace trace_with_training(True) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 416, in trace_with_training fn.get_concrete_function(*args, **kwargs) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 547, in get_concrete_function return super(LayerCall, self).get_concrete_function(*args, **kwargs) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 959, in get_concrete_function concrete = self._get_concrete_function_garbage_collected(*args, **kwargs) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 865, in _get_concrete_function_garbage_collected self._initialize(args, kwargs, add_initializers_to=initializers) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 506, in _initialize *args, **kwds)) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 2446, in _get_concrete_function_internal_garbage_collected graph_function, _, _ = self._maybe_define_function(args, kwargs) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 2777, in _maybe_define_function graph_function = self._create_graph_function(args, kwargs) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 2667, in _create_graph_function capture_by_value=self._capture_by_value), File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/framework/func_graph.py", line 981, in func_graph_from_py_func func_outputs = python_func(*func_args, **func_kwargs) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 441, in wrapped_fn return weak_wrapped_fn().__wrapped__(*args, **kwds) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 524, in wrapper ret = method(*args, **kwargs) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/utils.py", line 170, in wrap_with_training_arg lambda: replace_training_and_call(False)) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/utils/tf_utils.py", line 65, in smart_cond pred, true_fn=true_fn, false_fn=false_fn, name=name) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/framework/smart_cond.py", line 54, in smart_cond return true_fn() File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/utils.py", line 169, in <lambda> lambda: replace_training_and_call(True), File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/utils.py", line 165, in replace_training_and_call return wrapped_call(*args, **kwargs) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 566, in call_and_return_conditional_losses return layer_call(inputs, *args, **kwargs), layer.get_losses_for(inputs) File "/home/tensorflow/.local/lib/python3.6/site-packages/object_detection/meta_architectures/ssd_meta_arch.py", line 251, in call return self._extract_features(inputs) File "/home/tensorflow/.local/lib/python3.6/site-packages/object_detection/models/ssd_efficientnet_bifpn_feature_extractor.py", line 224, in _extract_features list(zip(self._output_layer_alias, base_feature_maps))) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 968, in __call__ outputs = self.call(cast_inputs, *args, **kwargs) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/utils.py", line 71, in return_outputs_and_add_losses outputs, losses = fn(inputs, *args, **kwargs) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/utils.py", line 170, in wrap_with_training_arg lambda: replace_training_and_call(False)) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/utils/tf_utils.py", line 65, in smart_cond pred, true_fn=true_fn, false_fn=false_fn, name=name) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/framework/smart_cond.py", line 54, in smart_cond return true_fn() File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/utils.py", line 169, in <lambda> lambda: replace_training_and_call(True), File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/utils.py", line 165, in replace_training_and_call return wrapped_call(*args, **kwargs) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 541, in __call__ self.call_collection.add_trace(*args, **kwargs) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 418, in add_trace trace_with_training(True) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 416, in trace_with_training fn.get_concrete_function(*args, **kwargs) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 547, in get_concrete_function return super(LayerCall, self).get_concrete_function(*args, **kwargs) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 959, in get_concrete_function concrete = self._get_concrete_function_garbage_collected(*args, **kwargs) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 865, in _get_concrete_function_garbage_collected self._initialize(args, kwargs, add_initializers_to=initializers) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 506, in _initialize *args, **kwds)) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 2446, in _get_concrete_function_internal_garbage_collected graph_function, _, _ = self._maybe_define_function(args, kwargs) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 2777, in _maybe_define_function graph_function = self._create_graph_function(args, kwargs) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 2667, in _create_graph_function capture_by_value=self._capture_by_value), File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/framework/func_graph.py", line 981, in func_graph_from_py_func func_outputs = python_func(*func_args, **func_kwargs) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 441, in wrapped_fn return weak_wrapped_fn().__wrapped__(*args, **kwds) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 524, in wrapper ret = method(*args, **kwargs) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 484, in wrap_with_training_arg return call_fn(*args, **kwargs) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save_impl.py", line 566, in call_and_return_conditional_losses return layer_call(inputs, *args, **kwargs), layer.get_losses_for(inputs) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 1627, in get_losses_for reachable = tf_utils.get_reachable_from_inputs(inputs, losses) File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/keras/utils/tf_utils.py", line 140, in get_reachable_from_inputs raise TypeError('Expected Operation, Variable, or Tensor, got ' + str(x)) TypeError: Expected Operation, Variable, or Tensor, got level_5 6. System information
- OS Platform and Distribution : Linux Ubuntu 16.04, docker 19.3, nvidia-docker
- Mobile device name if the issue happens on a mobile device: N/A
- TensorFlow installed from (source or binary): Docker image, binary
- TensorFlow version (use command below): 2.2.0
- Python version: 3.6
- Bazel version (if compiling from source):N/A
- GCC/Compiler version (if compiling from source):N/A
- CUDA/cuDNN version: N/A
- GPU model and memory: N/A
v2.2.0-rc4-8-g2b96f3662b 2.2.0