hevc: clip pixels when transquant bypass is used
authorGuillaume Martres <smarter@ubuntu.com>
Sat, 11 Jan 2014 21:46:24 +0000 (22:46 +0100)
committerMichael Niedermayer <michaelni@gmx.at>
Sat, 11 Jan 2014 23:49:11 +0000 (00:49 +0100)
commitc9fe0caf7a1abde7ca0b1a359f551103064867b1
tree5e2daab15f04d19bd0315ac5696b68bf3f532289
parent65801040c648bbe352ad48ced2045bbeb25e9fcf
hevc: clip pixels when transquant bypass is used

Fixes: asan_stack-oob_eae8e3_7333_WPP_B_ericsson_MAIN10_2.bit
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
This is a more proper fix than 5856bca360c5bc3e340a357d91b1f993c80a7bea

The reconstructed picture should always be clipped (see section 8.6.5),
previously we did not clip coding units where
cu_transquant_bypass_flag == 1

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
libavcodec/hevcdsp_template.c