avcodec/ffv1enc: Fix out-of-bounds-array access
authorAndreas Rheinhardt <andreas.rheinhardt@gmail.com>
Sat, 28 Sep 2019 02:26:01 +0000 (04:26 +0200)
committerMichael Niedermayer <michael@niedermayer.cc>
Sat, 28 Sep 2019 15:24:32 +0000 (17:24 +0200)
commitf7bc0386d935e541a610a36595476710c036bfae
tree6bf8d3bbb39dbb8e6b8f45e226399f99ee362192
parent736c7c20e7819811dc59f43490563789b192eb6e
avcodec/ffv1enc: Fix out-of-bounds-array access

libavcodec/ffv1enc.c accessed an array of uint8_t [32] via array[0][j]
in order to loop over all the uint8_t in this array of arrays. Of course
this implied an out-of-bounds access for array[0] and UBSan complained
about this. So replace this with nested loops; furthermore, factor this
out into a function of its own to easily break out of the nested loops.

This affected the FATE-tests vsynth1-ffv1, vsynth1-ffv1-v3-yuv420p,
vsynth1-ffv1-v3-yuv422p10, vsynth1-ffv1-v3-yuv444p16,
vsynth1-ffv1-v3-bgr0, vsynth1-ffv1-ffv1-v3-rgb48 as well as the
corresponding vsynth2-*, vsynth3-* and the vsynth_lena-* tests.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
libavcodec/ffv1enc.c