use vImage framework to accelerate conversion from YUV to RGB(A)

[ledyba-z]

Hi. I am working at Link-U, a company which publishes digital comics in Japan.

In this pull request, I added an optimized code path to speed up conversions from YUV to RGB(A) using vImage framework.

Unfortunately, it seems that vImage conversion API does not provide fully support for 10bit/12bit YUV image. Hence, in this patch, conversions from only 8bit YUV images are accelerated.

Benchmark result

All results are measured in iPhone 6, iOS 12.2.

Original version

Here is a branch with measuring code.

# 8-bit image SDWebImageAVIFCoder_Example[765:156941] Converting from YUV to RGB takes 0.174398 sec SDWebImageAVIFCoder_Example[765:156924] Static AVIF load success # 10-bit image SDWebImageAVIFCoder_Example[765:156941] Converting from YUV to RGB takes 0.465885 sec SDWebImageAVIFCoder_Example[765:156924] HDR AVIF load success 

vImage version(this patch)

Here is a branch with measuring code.

# 8-bit image SDWebImageAVIFCoder_Example[768:157661] Converting from YUV to RGB takes 0.007166 sec SDWebImageAVIFCoder_Example[768:157548] Static AVIF load success # 10-bit image SDWebImageAVIFCoder_Example[768:157661] Converting from YUV to RGB takes 0.460658 sec SDWebImageAVIFCoder_Example[768:157548] HDR AVIF load success 

This patch makes a conversion more than 24 times faster, in 8-bit image.

Please take a look!

[ledyba-z]

P.S.

Monochrome images are also considered in this patch, however, libavif does not handle them correctly.

I sent a pull request about this issue to libavif:
AOMediaCodec/libavif#39

[dreampiggy]

Seems good...
Previouslly, to support HDR or 12bit AVIF image (I test it using the https://github.com/AOMediaCodec/av1-avif/tree/master/testFiles), I have to use the CPU code to do this :)

If the vImage can have a better performance, I'd like to replace that.

[ledyba-z]

Thank you!

For example, there is no conversion API from 10bit/12bit YUV420 to RGB888.
However, there can be some alternative way to convert (for example, converting 10bit YUV420 → 10bit YUV444 → 8bit RGB or 16bit RGB).
I will try them.

By the way, what should 10-bit/12-bit images be converted to?
Currently, 10bit/12bit images also seem to be converted to 8bit RGB(A):

SDWebImageAVIFCoder/SDWebImageAVIFCoder/Classes/SDImageAVIFCoder.m

Line 164 in ca147aa

CGImageRef imageRef = CGImageCreate(width, height, bitsPerComponent, bitsPerPixel, rowBytes, colorSpaceRef, bitmapInfo, provider, NULL, NO, renderingIntent);

How about converting them to 16bit RGB(A)?

[ledyba-z]

@dreampiggy
(Sorry, I accidentally close PR...)

Hi, I succeeded to support 10bit/12bit images.

benchmark @ iPhone 6 # before SDWebImageAVIFCoder_Example[768:157661] Converting from YUV to RGB takes 0.460658 sec SDWebImageAVIFCoder_Example[768:157548] HDR AVIF load success # after SDWebImageAVIFCoder_Example[713:130564] Converting from YUV to RGB takes 0.090242 sec SDWebImageAVIFCoder_Example[713:130433] HDR AVIF load success 

10bit/12bit versions seems not strongly optimized than 8bit versions yet, but it makes about 5 times faster!

By the way, test images in some format are not available at official repo:

• 8bit/10bit/12bit YUV422
• 10bit/12bit monochrome

So, I prepared some examples images by myself using cavif, so please use them to check:

https://github.com/link-u/avif-sample-images

Please take a look!

[dreampiggy]

I’ll test it soon. Thanks for your patient.

I’m in China and on vacation in China’s lunar new year. If I have a time to check, I’ll merge and release a new version.

[dreampiggy]

Please merge the master branch again. I fixed the Travis-CI issue and support SwiftPM now. You can check whether this dev branch works on different compile case. (For example, SwiftPM/Carthage does not have dav1d or rav1e, always use libavif method and don’t directly use them)

[dreampiggy]

@ledyba-z libavif released new v0.5.4, does that fix the monochrome issue ?

https://github.com/AOMediaCodec/libavif/blob/master/CHANGELOG.md

AOMediaCodec/libavif@e080130

[ledyba-z]

@dreampiggy Thanks, please enjoy 春节!

I merged master branch to this branch, so Circle CI will check it...

[ledyba-z]

@ledyba-z libavif released new v0.5.4, does that fix the monochrome issue ?
https://github.com/AOMediaCodec/libavif/blob/master/CHANGELOG.md
AOMediaCodec/libavif@e080130

Yeah, applied patch was the same I sent, so the monochrome issue shall be fixed.
You can compare the result between the original version and vImage version(this PR) also in monochrome images.

[dreampiggy]

Good. I’ve already remove that bytesNormal check. it’s time to merge now ? Anything else pending ?

[ledyba-z]

@dreampiggy Thank you!
I think it's okay to merge.

[dreampiggy]

@ledyba-z Hi. The 0.5.0 version is released, check: v0.5.0 changelog for details.

Thanks for your support, you're actually the first contributor here, I've updated the Readme with this :)

[dreampiggy]

I'm decided to support thumbnail decoding for AVIF soon in v0.6.0 (mentioned in SDWebImage 5.5.0 new feature, which need each codec update to support). Like the similiar update to WebPCoder.

If you have some better idea about this, please contact me, thanks.

[dreampiggy]

One possible issue I don't actually realized. The libavif 0.5.0 provide the API to grab the EXIF/XMP metadata (See AOMediaCodec/libavif#26). Does AVIF support EXIF orientation like JPEG ?

If so, maybe we should update the code to provide a fix for the non-Up AVIF images.

[ledyba-z]

@dreampiggy Thank you, you are welcome!

support thumbnail decoding for AVIF

MIAF, a file format for AVIF and HEIF, can hold thumbnail images (it's called MIAF thumbnail images).

However, there seems not to be any example files with thumbnails yet, and libavif might not support decoding thumbnail images yet.

Does AVIF support EXIF orientation like JPEG ?

YES, but it's not need to parse EXIF.

AVIF supports orientation/mirroing/crop itself. And also, MIAF specifies that readers are not needed to parse exif metadata to rotate/mirror/crop (If you want to read other metadata, you still have to read exif).

I prepared the test files, so please check our example files.

Our encoder, cavif, and decoder, davif, supports those features. But libavif looks not support them at all. I will report an issue.