libopenjpegenc: fix out-of-bounds reads when filling the edges
authorAndreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
Thu, 13 Oct 2016 20:14:46 +0000 (22:14 +0200)
committerAndreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
Mon, 17 Oct 2016 15:21:35 +0000 (17:21 +0200)
commitd391719be19be2f2716dcb1da9f88b0b4214e4c4
treea7ffd94a0f94e7cc3918d6b4a6833e3c26019ec3
parenta22155dacd935e745c439a8646626c13f7e6aae4
libopenjpegenc: fix out-of-bounds reads when filling the edges

The calculation of width/height should round up, not round down to
prevent setting width or height to 0.

Also image->comps[compno].w is unsigned (at least in openjpeg2), so the
calculation could silently wrap around without the explicit cast to int.

Reviewed-by: Michael Bradshaw <mjbshaw@gmail.com>
Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
(cherry picked from commit 56706ac0d5723cb549fec2602e798ab1bf6004cd)
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
libavcodec/libopenjpegenc.c