avcodec/mlpdsp: Fix signed integer overflow, 2nd try
authorMichael Niedermayer <michael@niedermayer.cc>
Mon, 20 Nov 2017 17:45:45 +0000 (18:45 +0100)
committerMichael Niedermayer <michael@niedermayer.cc>
Sat, 30 Dec 2017 20:11:32 +0000 (21:11 +0100)
commitcead6c94c502a90f1318ddc47885bfaa407068dd
treec5bcc40ff2b152f56c1e50ea4984ae855f28f65f
parentd5bdcd8a2702623af06a0db76a7f7f68f08c457b
avcodec/mlpdsp: Fix signed integer overflow, 2nd try

The outputted bits should match what is used in the lossless check

Fixes: runtime error: signed integer overflow: -538697856 * 256 cannot be represented in type 'int'
Fixes: 4326/clusterfuzz-testcase-minimized-5689449645080576

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 97c00edaa043043c29d985653e7e1687b56dfa23)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
libavcodec/mlpdsp.c