avcodec/pixlet: fixes integer overflow in read_highpass()
authorMichael Niedermayer <michael@niedermayer.cc>
Thu, 17 Aug 2017 01:54:56 +0000 (03:54 +0200)
committerMichael Niedermayer <michael@niedermayer.cc>
Fri, 18 Aug 2017 09:34:01 +0000 (11:34 +0200)
Fixes: runtime error: negation of -2147483648 cannot be represented in type 'int32_t' (aka 'int'); cast to an unsigned type to negate this value to itself
Fixes: 2879/clusterfuzz-testcase-minimized-6317542639403008

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
libavcodec/pixlet.c

index a9661d3..088226b 100644 (file)
@@ -331,6 +331,9 @@ static int read_highpass(AVCodecContext *avctx, uint8_t *ptr, int plane, AVFrame
             return AVERROR_INVALIDDATA;
         }
 
+        if (a == INT32_MIN)
+            return AVERROR_INVALIDDATA;
+
         ret = read_high_coeffs(avctx, ptr + bytestream2_tell(&ctx->gb), dest, size,
                                c, (b >= FFABS(a)) ? b : a, d,
                                ctx->band[plane][i + 1].width, stride);