difference mAP with kINT8 Calibration

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

From deepstream-plugin project I run YOLOV2 detector network with kINT8 calibration that works OK. I delete the original calibration table file (sources/lib/calibration/yolov2-calibration.table) and fill some validation sample to data/calibration_images.txt file for the yolov2 calibration table re-generation. It encountered errors and failed to build TensorRT engine.

./sources/apps/trt-yolo/trt-yolo-app writeCalibrationCache sources/lib/calibration/yolov2-calibration.table trt-yolo-app: NvPluginYOLO.cu:58: virtual void nvinfer1::plugin::PReLU::configure(const nvinfer1::Dims*, int, const nvinfer1::Dims*, int, int): Assertion `mBatchDim == 1' failed. Aborted (core dumped) -- re-launch trt-yolo-app Building the TensorRT Engine... readCalibrationCache from file: sources/lib/calibration/yolov2-calibration.table Using cached calibration table to build the engine Building complete! Serializing the TensorRT Engine... Serialized plan file cached at location : sources/lib/models/yolov2-kINT8-batch1.engine Loading TRT Engine... Loading Complete!

The TensorRT engine was generated successfully after re-launch trt-yolo-app. But the content of calibration table is difference with original. I test kIN8 mAP for model. It show 1 or 2 percent difference very calibration table re-generation. Is it a known symptom or not?

The test env: TensorRT-4.0.1.6-cuda-9.2, 1080ti, Ubuntu 16.0.4LTS with 4.15.0-43-generic kernel. Here are a sample of calibration table.

1 (Unnamed ITensor* 52): 3cdc9e53 (Unnamed ITensor* 44): 3d3ba1be (Unnamed ITensor* 38): 3d5e9ecc (Unnamed ITensor* 49): 3ce41530 (Unnamed ITensor* 20): 3d65bc0b (Unnamed ITensor* 35): 3d48d9cd (Unnamed ITensor* 55): 3cd89b86 (Unnamed ITensor* 41): 3d496fe9 (Unnamed ITensor* 33): 3d03ed95 (Unnamed ITensor* 30): 3d21ed83 (Unnamed ITensor* 5): 3cc08982 (Unnamed ITensor* 2): 3c008912 (Unnamed ITensor* 54): 3d088782 (Unnamed ITensor* 25): 3d33360b (Unnamed ITensor* 1): 4000890a (Unnamed ITensor* 13): 3d2c1de4 (Unnamed ITensor* 67): 3cfc4eb0 (Unnamed ITensor* 32): 3d4c2587 (Unnamed ITensor* 18): 3d30d396 (Unnamed ITensor* 23): 3d20e995 (Unnamed ITensor* 22): 3d3e3c98 (Unnamed ITensor* 16): 3d1ce3b0 (Unnamed ITensor* 46): 3d01f73f (Unnamed ITensor* 48): 3d0b0fcc (Unnamed ITensor* 63): 3d115915 (Unnamed ITensor* 71): 3ce024bc (Unnamed ITensor* 64): 3cefa77f (Unnamed ITensor* 29): 3d21ed83 (Unnamed ITensor* 26): 3d1f1ee3 data: 40000889 (Unnamed ITensor* 66): 3d1d45a7 (Unnamed ITensor* 57): 3d0419dd (Unnamed ITensor* 60): 3d028be7 (Unnamed ITensor* 4): 3cf1d39f (Unnamed ITensor* 61): 3cd94e6a (Unnamed ITensor* 58): 3cd11ab5 (Unnamed ITensor* 9): 3d1432a4 (Unnamed ITensor* 51): 3d0936c4 (Unnamed ITensor* 75): 3d0973dc (Unnamed ITensor* 19): 3d65bc0b (Unnamed ITensor* 71) copy: 3ce024bc (Unnamed ITensor* 67) copy: 3cfc4eb0 region_32: 3c011169 (Unnamed ITensor* 74): 3d2bdc55 (Unnamed ITensor* 70): 3ce024bc (Unnamed ITensor* 42): 3d008114 (Unnamed ITensor* 69): 3d30262a (Unnamed ITensor* 36): 3d04316d (Unnamed ITensor* 28): 3d6b5bde (Unnamed ITensor* 39): 3d004284 (Unnamed ITensor* 6): 3d3d3327 (Unnamed ITensor* 8): 3d767c7d (Unnamed ITensor* 15): 3d30d42e (Unnamed ITensor* 45): 3ce945a0 (Unnamed ITensor* 76): 3decdf3d (Unnamed ITensor* 12): 3d740386 (Unnamed ITensor* 10): 3d45a2e7
2
  1. The “Assertion” problem is a known problem. It will be fixed in tensorRT 5.1 version.
  2. 2 factors affect calibration table
    a. The distribution of calibration images
    b. Different tensorRT version may have different calibration algorithm.