avcodec/rangecoder: Test for invalid corner case
authorMichael Niedermayer <michael@niedermayer.cc>
Mon, 8 May 2017 00:58:54 +0000 (02:58 +0200)
committerMichael Niedermayer <michael@niedermayer.cc>
Mon, 8 May 2017 01:25:17 +0000 (03:25 +0200)
Fixes runtime error: left shift of 1912602815 by 8 places cannot be represented in type 'int'
Fixes: 1403/clusterfuzz-testcase-minimized-4724820484816896

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/targets/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
libavcodec/rangecoder.c

index 9c6ef75..d7a2774 100644 (file)
@@ -58,6 +58,10 @@ av_cold void ff_init_range_decoder(RangeCoder *c, const uint8_t *buf,
 
     c->low         = AV_RB16(c->bytestream);
     c->bytestream += 2;
+    if (c->low >= 0xFF00) {
+        c->low = 0xFF00;
+        c->bytestream_end = c->bytestream + 2;
+    }
 }
 
 void ff_build_rac_states(RangeCoder *c, int factor, int max_p)