TAO 5.0.0. TF1 Container fail to run tao model yolo_v4 dataset_convert command

[pramaniksandip]

Please provide the following information when requesting support.

• Hardware (T4/V100/Xavier/Nano/etc) A100
• Network Type SSD

I am using nvcr.io/nvidia/tao/tao-toolkit:5.0.0-tf1.15.5 container. The container is running successful in Kubernetes environment in DGX A100 in a GPU.

But following command though error.
root@tao-f865859d9-6scz9:/workspace# tao model yolo_v4 dataset_convert -d /workspace/nemo/data/nvidia-tao/tensorflow/yolo_v4/specs/yolo_v4_tfrecords_kitti_train.txt -o /workspace/nemo/data/kitti_data/train/tfrecords/train
~/.tao_mounts.json wasn’t found. Falling back to obtain mount points and docker configs from ~/.tao_mounts.json.
Please note that this will be deprecated going forward.
Traceback (most recent call last):
File “/usr/local/bin/tao”, line 8, in
sys.exit(main())
File “/usr/local/lib/python3.8/dist-packages/nvidia_tao_cli/entrypoint/tao_launcher.py”, line 134, in main
instance.launch_command(
File “/usr/local/lib/python3.8/dist-packages/nvidia_tao_cli/components/instance_handler/local_instance.py”, line 356, in launch_command
docker_logged_in(required_registry=task_map[task].docker_registry)
File “/usr/local/lib/python3.8/dist-packages/nvidia_tao_cli/components/instance_handler/utils.py”, line 151, in docker_logged_in
data = load_config_file(docker_config)
File “/usr/local/lib/python3.8/dist-packages/nvidia_tao_cli/components/instance_handler/utils.py”, line 84, in load_config_file
assert os.path.exists(config_path), (
AssertionError: Config path must be a valid unix path. No file found at: /root/.docker/config.json. Did you run docker login?
root@tao-f865859d9-6scz9:/workspace#

List of tao packages inside the POD
root@tao-f865859d9-6scz9:/workspace# pip list|grep tao
nvidia-eff-tao-encryption 0.1.7
nvidia-tao 5.0.0
nvidia-tao-tf1 5.0.0.151.dev0
root@tao-f865859d9-6scz9:/workspace#

[Morganh]

@pramaniksandip

root@tao-f865859d9-6scz9

How did you trigger this container? Could you share the full command?

[pramaniksandip]

@Morganh

Please find the command to trigger container,

apiVersion: apps/v1
kind: Deployment
metadata:
name: tao
spec:
selector:
matchLabels:
run: tao
replicas: 1
template:
metadata:
labels:
run: tao
spec:
nodeName: dgxa100
containers:
- name: tao
image: nvcr.io/nvidia/tao/tao-toolkit:5.0.0-tf1.15.5
command: [“/bin/sh”, “-c”]
args:
- jupyter lab --allow-root --ip=0.0.0.0 --NotebookApp.token=‘tao’;
resources:
limits:
nvidia.com/${profile}: 1
volumeMounts:
- name: tao-data
mountPath: /workspace/nemo/data
ports:
- containerPort: 8888
volumes:
- name: tao-data
hostPath:
path: /home/tao
type: Directory

[Morganh]

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

@pramaniksandip

nvcr.io/nvidia/tao/tao-toolkit:5.0.0-tf1.15.5

You already run inside the tao docker.
So, please run without tao model.
For example,
root@tao-f865859d9-6scz9:/workspace# yolo_v4 dataset_convert xxx