avcodec/ilbcdec: Fix multiple integer overflows
authorMichael Niedermayer <michael@niedermayer.cc>
Wed, 10 Oct 2018 23:04:34 +0000 (01:04 +0200)
committerMichael Niedermayer <michael@niedermayer.cc>
Thu, 18 Oct 2018 00:28:54 +0000 (02:28 +0200)
commitd7dbad12f820a72997772ff5e535bee3eb7c5b4b
treebea76a7d7bb4402fb1ae95280489b5a46ccdba58
parentd01788665767720441e2d506a270ee6ba85e9fe2
avcodec/ilbcdec: Fix multiple integer overflows

Fixes: 10651/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_ILBC_fuzzer-5202341540659200
Fixes: signed integer overflow: -1707705920 - 1703592888 cannot be represented in type 'int'

This tries to follow the webrtc code. For example using cliping and 64 bit as in WebRtcSpl_DotProductWithScale()
and not doing so in other places.
I could not find anything in rfc3951 and the reference code inside which would
explain what to do in these corner cases.

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
libavcodec/ilbcdec.c