IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming deployments

augraphics/pcs-logo Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming Vignesh V Menon, Hadi Amirpour, Christian Timmerer, Mohammad Ghanbari Christian Doppler Laboratory ATHENA, Institute of Information Technology (ITEC), University of Klagenfurt, Austria Date: 02 July 2021 Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 1

Outline 1 Introduction 2 State-of-the-art: Multi-Rate Encoding 3 State-of-the-art: Multi-Encoding 4 Efficient Multi-Rate Encoding 5 Efficient Multi-Encoding 6 Experimental Evaluation 7 Conclusions and Future Directions Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 2

Introduction Introduction Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 3

Introduction Introduction Background of HTTP Adaptive Streaming (HAS)1 Source: https://bitmovin.com/adaptive-streaming/ Why Adaptive Streaming? Adapt for a wide range of devices Adapt for a broad set of Internet speeds What HAS does? Each source video is split into segments Encoded at multiple bitrates, resolutions, and codecs Delivered to the client based on the device capability, network speed etc. 1 A. Bentaleb et al. “A Survey on Bitrate Adaptation Schemes for Streaming Media Over HTTP”. In: IEEE Communications Surveys Tutorials 21.1 (2019), pp. 562–585. doi: 10.1109/COMST.2018.2862938. Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 4

Introduction Introduction Encoding time of HAS representations Relative encoding time of the representations normalized to encoding time of the 2160p 25Mbps representation. 500 1000 1500 2000 3000 4500 5800 7000 11600 16800 20000 25000 Bitrate (in kbps) 0 20 40 60 80 100 Relative Time Complexity (in percentage) 540p 1080p 2160p As resolution doubles, encoding time complexity doubles! Many encoder analysis decisions are redundant across the representations. Multi-rate: Exploit this redundancy across representations of a resolution. Multi-resolution: Exploit this redundancy across resolutions. Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 5

State-of-the-art: Multi-Rate Encoding State-of-the-art: Multi-Rate Encoding Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 6

State-of-the-art: Multi-Rate Encoding Related Works: Multi-Rate Encoding Single-bound2 and Double-bound3 for CU depth estimation 3840x2160 @ 25Mbps 3840x2160 @ 11.6Mbps 3840x2160 @ 20Mbps 3840x2160 @ 16.8Mbps (a) Single-bound for CU depth estimation 3840x2160 @ 25Mbps 3840x2160 @ 11.6Mbps 3840x2160 @ 20Mbps 3840x2160 @ 16.8Mbps (b) Double-bound for CU depth estimation Motivation: CUs generally tend to have higher depth values (smaller CU sizes) in higher bitrate representations. CU depth information from the highest bitrate representation is used as the upper bound of the CU depth estimation of dependent representations. In double-bound, additionally, CU depth information from the lowest bitrate representation is used as the lower bound of the CU depth estimation of dependent representations. 2 D. Schroeder, P. Rehm, and E. Steinbach. “Block structure reuse for multi-rate high efficiency video coding”. In: 2015 IEEE International Conference on Image Processing (ICIP). Sept. 2015, pp. 3972–3976. 3 H. Amirpour et al. “Fast Multi-rate Encoding for Adaptive HTTP Streaming”. In: 2020 Data Compression Conference (DCC). 2020, pp. 358–358. Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 7

State-of-the-art: Multi-Rate Encoding Related Works: Multi-Rate Encoding Integrated x265 mode-1 and x265 mode-24 Integrated x265 mode-1 Force the optimal decisions made in the highest bitrate representation the slice-type scene-cut decisions the quadtree structure prediction modes reference indices Integrated x265 mode-2 Reuse all CUs and PUs from the highest bitrate representation without additional RDO. They also employ static refinement techniques defined in x265. If the reference depth is d, it is re- evaluated by using the RDO against the cost of splitting the CU (i.e., depth d + 1). The motion vectors from the highest bitrate representation are used as Motion vector predictors (MVPs). 4 K. Goswami et al. “Adaptive Multi-Resolution Encoding for ABR Streaming”. In: 2018 25th IEEE International Conference on Image Processing (ICIP). 2018, pp. 1008–1012. Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 8

State-of-the-art: Multi-Encoding State-of-the-art: Multi-Encoding Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 9

State-of-the-art: Multi-Encoding Related Work: Multi-Encoding 1920x1080 @ 7.0Mbps 1920x1080 @ 3.0Mbps 1920x1080 @ 5.8Mbps 1920x1080 @ 4.5Mbps 3840x2160 @ 25.0Mbps 3840x2160 @ 11.6Mbps 3840x2160 @ 20.0Mbps 3840x2160 @ 16.8Mbps Scaled CU,PU reuse + refinement Figure: Encoder analysis sharing method in x265 Algorithm.5 5 Aruna Mathesawaran et al. “Open source framework for reduced-complexity multi-rate HEVC encoding”. In: Applications of Digital Image Processing XLIII. ed. by Andrew G. Tescher and Touradj Ebrahimi. Vol. 11510. International Society for Optics and Photonics. SPIE, 2020, pp. 461 –471. Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 10

Efficient Multi-Rate Encoding Efficient Multi-Rate Encoding Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 11

Efficient Multi-Rate Encoding Efficient Multi-Rate Encoding Prediction Mode Heuristics Prediction Mode Heuristics If the CU depth chosen for the dependent representations is the same as the reference representation, the following heuristics are proposed. If Skip mode was chosen in the highest bitrate representation, RDO is evaluated for only Merge/Skip modes. If 2Nx2N mode was chosen in the highest bitrate representation, skip RDO for AMP modes. If inter-prediction mode was chosen in the highest bitrate representation, skip RDO for intra-prediction modes. In double-bound for CU estimation, if the intra-prediction mode was chosen for the highest and lowest bitrate representations, RDO is evaluated for only intra-prediction modes in the intermediate representations. Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 12

Efficient Multi-Rate Encoding Efficient Multi-Rate Encoding Motion Estimation Heuristics Motion Estimation Heuristics If the CU depth and PU chosen for the dependent representations is the same as the reference representation, the following heuristics are proposed. Force the same reference frame as that of the highest bitrate representation. The Motion Vector Predictor (MVP) is set to be the Motion Vector (MV) of the highest bitrate representation. In double-bound for CU estimation, the motion search range is decreased to a smaller window if the MV of the highest bitrate representation and the MV of the lowest bitrate representation are close to each other. The search range is determined to be the maximum difference between the x coordinates and y coordinates of the MVs. Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 13

Efficient Multi-Rate Encoding Efficient Multi-Rate Encoding Proposed Multi-rate Algorithm-1 and Algorithm-2 3840x2160 @ 25Mbps 3840x2160 @ 11.6Mbps 3840x2160 @ 20Mbps 3840x2160 @ 16.8Mbps (a) Proposed Multi-rate Algorithm-1 3840x2160 @ 25Mbps 3840x2160 @ 11.6Mbps 3840x2160 @ 20Mbps 3840x2160 @ 16.8Mbps (b) Proposed Multi-rate Algorithm-2 − > denotes analysis sharing using single-bound for CU depth estimation and proposed heuristics. − > denotes analysis sharing using double-bound for CU depth estimation and proposed heuristics. Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 14

Efficient Multi-Encoding Efficient Multi-Encoding Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 15

Efficient Multi-Encoding Efficient Multi-Encoding Proposed Multi-Encoding Algorithm-1 1920x1080 @ 7.0Mbps 1920x1080 @ 3.0Mbps 1920x1080 @ 5.8Mbps 1920x1080 @ 4.5Mbps 3840x2160 @ 25.0Mbps 3840x2160 @ 11.6Mbps 3840x2160 @ 20.0Mbps 3840x2160 @ 16.8Mbps Figure: Encoder analysis sharing method in Proposed Multi-Encoding Algorithm-1. Multi-rate Approach: Proposed Multi-rate Algorithm-2 Multi-resolution Approach: scaled CU information from highest bitrate representation of the previous resolution is used as lower bound of CU depth estimation process. Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 16

Efficient Multi-Encoding Efficient Multi-Encoding Proposed Multi-Encoding Algorithm-2 1920x1080 @ 7.0Mbps 1920x1080 @ 3.0Mbps 1920x1080 @ 5.8Mbps 1920x1080 @ 4.5Mbps 3840x2160 @ 25.0Mbps 3840x2160 @ 11.6Mbps 3840x2160 @ 20.0Mbps 3840x2160 @ 16.8Mbps Scaling + Single-bound Figure: Encoder analysis sharing method in Proposed Multi-Encoding Algorithm-2. Multi-rate Approach: Proposed Multi-rate Algorithm-2 Multi-resolution Approach: scaled CU information from lowest bitrate representation of the previous resolution is used as lower bound of CU depth estimation process. More rigid bound compared to Multi-encoding Algorithm-1. Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 17

Experimental Evaluation Experimental Evaluation Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 18

Experimental Evaluation Experimental Evaluation Test Setup Algorithms presented here are tested using x265 v3.46 with the veryslow preset (highest quality preset practically usable of x265). Test PC: Dual-processor server with Intel Xeon Gold 5218R (80 cores, frequency at 2.10 GHz). Test Sequences: (i) 4 JVET test sequences7 and (ii) 3 professionally produced UltraHD HDR cinematic content.8 The lowest resolution is 960 × 540, intermediate resolution is 1920 × 1080, and the highest resolution is 3840 × 2160. The lower resolution sources were generated from the original video source by applying bi-cubic scaling using FFmpeg.9 6 MulticoreWare Inc. x265 HEVC Encoder/H.265 Video Codec. url: http://x265.org/. 7 Jill Boyce et al. JVET-J1010: JVET common test conditions and software reference configurations. July 2018. 8 M. H. Pinson. “The Consumer Digital Video Library [Best of the Web]”. In: IEEE Signal Processing Magazine 30.4 (2013), pp. 172–174. doi: 10.1109/MSP.2013.2258265. 9 FFmpeg. FFmpeg Documentation. url: https://ffmpeg.org/ffmpeg.html. Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 19

Experimental Evaluation Experimental Evaluation Test Setup Table: Bitrate representations Bi,j (in kbps) based on HLS authoring specification10 Layer (i) Resolution j = 1 j = 2 j = 3 j = 4 1 960x540 2000 1500 1000 500 2 1920x1080 7000 5800 4500 3000 3 3840x2160 25000 20000 16800 11600 Performance evaluation Metrics: ∆T : overall encoding time saving BDRP : Average difference in bitrate with respect to reference encode to maintain the same PSNR BDRV : Average difference in bitrate with respect to reference encode to maintain the same VMAF BDRP ∆T : Trade-off between BDR and ∆T. 10 https://developer.apple.com/documentation/http_live_streaming/hls_authoring_specification_for_apple_devices Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 20

Experimental Evaluation Experimental Evaluation Multi-rate Encoding Table: Results for the multi-rate algorithms Method ∆T BDRP BDRP ∆T BDRV BDRV ∆T Single-bound for CU estimation 14.50% -0.59% -4.39% 0.10% 1.08% Double-bound for CU estimation 26.37% 0.73% 3.44% 1.25% 5.54% x265 Algorithm-1 18.61% 1.67% 9.52% 1.56% 8.85% x265 Algorithm-2 55.11% 8.42% 15.62% 8.72% 16.10% Multi-rate Algorithm-1 (ours) 26.50% -0.28% -0.84% 0.43% 2.31% Multi-rate Algorithm-2 (ours) 37.37% 1.06% 3.36% 1.57% 4.88% Multi-rate Algorithm-1 improves ∆T of Single-bound for CU estimation by 12% with negligible increase in BDR. Multi-rate Algorithm-2 improves ∆T of Double-bound for CU estimation by 11% with negligible increase in BDR. Better BDR ∆T metric Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 21

Experimental Evaluation Experimental Evaluation Multi-Encoding Scheme ∆T BDRP BDRV State-of-the-art scheme 80.05% 13.53% 9.59% Multi-encoding Scheme-1 39.72% 2.32% 1.55% Multi-encoding Scheme-2 50.90% 3.45% 2.63% State-of-the-art Multi-encoding Algorithm-1 Multi-encoding Algorithm-2 0 2 4 6 8 10 12 14 16 BDR/ T (in percentage) BDRP/ T BDRV/ T State-of-the-art algorithm yields the highest ∆T. Multi-encoding Algorithm-1 has the lowest increase in BDR and the lowest BDR ∆T metric. Multi-encoding Algorithm-2 improves ∆T of Multi-encoding Algorithm-1 by 11% with negligible increase in BDR. Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 22

Conclusions and Future Directions Conclusions and Future Directions Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 23

Conclusions and Future Directions Conclusions and Future Directions Conclusions Presented efficient multi-encoding algorithms for HTTP Adaptive Streaming, tested with the open-source x265 HEVC encoder. Multi-rate Encoding: Proposed heuristics improved the encoding speed of the single-bound and double-bound for CU estimation by an average of 12% and 11%, respectively with a negligible increase in BD-Rate. BDR ∆T metric for proposed algorithms is better compared to the state-of-the-art algorithms. Multi-encoding: Proposed two algorithms that extended the proposed multi-rate encoding algorithms across resolutions. BDR ∆T metric for proposed algorithms are better compared to the state-of-the-art algorithm. Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 24

Conclusions and Future Directions Conclusions and Future Directions Future Directions The encoding speedup can be improved by parallelizing the execution of encoding instances. The dependent representations can be started after a delay of one frame execution time of the reference representation. Further work is required to extend the algorithms to resolutions, which are non-integer power-of-2 multiples of each other. Further work is required to investigate more recent codecs, e.g., VVC. Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 25

Conclusions and Future Directions Q & A Thank you for your attention! Vignesh V Menon (vignesh.menon@aau.at) Hadi Amirpour (hadi.amirpourazarian@aau.at) Christian Timmerer (Christian.Timmerer@aau.at) Mohammad Ghanbari (ghan@essex.ac.uk) Vignesh V Menon Efficient Multi-Encoding Algorithms for HTTP Adaptive Bitrate Streaming 26

IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming deployments

More Related Content

Similar to IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming deployments

More from Vignesh V Menon

Recently uploaded

IEEE PCS'21: Efficient multi-encoding for large-scale HTTP Adaptive Streaming deployments