avcodec/apedec: Make coeffsA/B uint32_t, this avoids several cases of undefined behavior
authorMichael Niedermayer <michael@niedermayer.cc>
Tue, 2 Jul 2019 10:30:32 +0000 (12:30 +0200)
committerMichael Niedermayer <michael@niedermayer.cc>
Wed, 31 Jul 2019 18:35:07 +0000 (20:35 +0200)
commit1aad8937f73f9c8cc337e3173402f47175d077e7
tree467bb84164be5d538e3ede9e49ffca5a02cfc55f
parentbf778af1493b0814696307432763246fb53c75e7
avcodec/apedec: Make coeffsA/B uint32_t, this avoids several cases of undefined behavior

Changing the type to an unsigned one to avoid many casts was suggested
This may be inadequate for fixing the UB on ILP64

Fixes: signed integer overflow: -1418162611 * 383 cannot be represented in type 'int'
Fixes: 15547/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_APE_fuzzer-5691384901664768

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
libavcodec/apedec.c