avcodec/hevcdsp_template.c: Fix undefined shift in FUNC(dequant)
authorMichael Niedermayer <michael@niedermayer.cc>
Fri, 22 Dec 2017 02:12:03 +0000 (03:12 +0100)
committerMichael Niedermayer <michael@niedermayer.cc>
Mon, 19 Feb 2018 01:40:54 +0000 (02:40 +0100)
Fixes: runtime error: left shift of negative value -180
Fixes: 4626/clusterfuzz-testcase-minimized-5647837887987712

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 0c9ab5ef9c1ee852c80c859c9e07efe8730b57ed)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
libavcodec/hevcdsp_template.c

index 57d18bd..4d2e6bf 100644 (file)
@@ -125,7 +125,7 @@ static void FUNC(transform_skip)(int16_t *_coeffs, int16_t log2_size)
     } else {
         for (y = 0; y < size; y++) {
             for (x = 0; x < size; x++) {
-                *coeffs = *coeffs << -shift;
+                *coeffs = *(uint16_t*)coeffs << -shift;
                 coeffs++;
             }
         }