avcodec/alac: fix undefined behavior with INT_MIN in lpc_prediction()
authorMichael Niedermayer <michael@niedermayer.cc>
Fri, 6 Sep 2019 08:36:43 +0000 (10:36 +0200)
committerMichael Niedermayer <michael@niedermayer.cc>
Tue, 24 Sep 2019 15:53:39 +0000 (17:53 +0200)
Fixes: signed integer overflow: -2147483648 * -1 cannot be represented in type 'int'
Fixes: 16786/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_ALAC_fuzzer-5632818851348480

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
libavcodec/alac.c

index 782d461..c606f2a 100644 (file)
@@ -222,7 +222,7 @@ static void lpc_prediction(int32_t *error_buffer, uint32_t *buffer_out,
         /* adapt LPC coefficients */
         error_sign = sign_only(error_val);
         if (error_sign) {
-            for (j = 0; j < lpc_order && (int)error_val * error_sign > 0; j++) {
+            for (j = 0; j < lpc_order && (int)(error_val * error_sign) > 0; j++) {
                 int sign;
                 val  = d - pred[j];
                 sign = sign_only(val) * error_sign;