swscale/utils: Fix invalid left shifts of negative numbers
authorAndreas Rheinhardt <andreas.rheinhardt@gmail.com>
Sat, 28 Sep 2019 02:26:02 +0000 (04:26 +0200)
committerMichael Niedermayer <michael@niedermayer.cc>
Sat, 28 Sep 2019 15:24:32 +0000 (17:24 +0200)
Affected the FATE-tests vsynth_lena-dv-411, vsynth1-dv-411,
vsynth2-dv-411 and hevc-paramchange-yuv420p.yuv420p10.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
libswscale/utils.c

index 1b1f779..57c4fd2 100644 (file)
@@ -390,7 +390,7 @@ static av_cold int initFilter(int16_t **outFilter, int32_t **filterPos,
             (*filterPos)[i] = xx;
             // bilinear upscale / linear interpolate / area averaging
             for (j = 0; j < filterSize; j++) {
-                int64_t coeff= fone - FFABS(((int64_t)xx<<16) - xDstInSrc)*(fone>>16);
+                int64_t coeff = fone - FFABS((int64_t)xx * (1 << 16) - xDstInSrc) * (fone >> 16);
                 if (coeff < 0)
                     coeff = 0;
                 filter[i * filterSize + j] = coeff;