avcodec/mlpdsp: Fix signed integer overflow, 2nd try
authorMichael Niedermayer <michael@niedermayer.cc>
Mon, 20 Nov 2017 17:45:45 +0000 (18:45 +0100)
committerMichael Niedermayer <michael@niedermayer.cc>
Wed, 31 Jan 2018 21:56:14 +0000 (22:56 +0100)
commit6011422a5415ee53ba195ffaa8d60f44e06cdbdf
tree42723e17bf3a3b02489bc157f3c346a91ce94900
parent32a92a7a9b43db2bcd6f4b6234d9070397957378
avcodec/mlpdsp: Fix signed integer overflow, 2nd try

The outputted bits should match what is used in the lossless check

Fixes: runtime error: signed integer overflow: -538697856 * 256 cannot be represented in type 'int'
Fixes: 4326/clusterfuzz-testcase-minimized-5689449645080576

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 97c00edaa043043c29d985653e7e1687b56dfa23)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
libavcodec/mlpdsp.c