avcodec/mlpdsp: Fix signed integer overflow, 2nd try
authorMichael Niedermayer <michael@niedermayer.cc>
Mon, 20 Nov 2017 17:45:45 +0000 (18:45 +0100)
committerMichael Niedermayer <michael@niedermayer.cc>
Fri, 1 Dec 2017 23:28:59 +0000 (00:28 +0100)
commit78b6e8fb233f52284d6a3bee169361f5ee82c79f
treebccc5ce9aae567eefb6c004422d7c129b06b7bc1
parent71e40180cb1df511cfb67e17ff041f69d8d11d32
avcodec/mlpdsp: Fix signed integer overflow, 2nd try

The outputted bits should match what is used in the lossless check

Fixes: runtime error: signed integer overflow: -538697856 * 256 cannot be represented in type 'int'
Fixes: 4326/clusterfuzz-testcase-minimized-5689449645080576

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 97c00edaa043043c29d985653e7e1687b56dfa23)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
libavcodec/mlpdsp.c