Fixes avpicture_layout to not write past buffer end.
authorMatthew Einhorn <moiein2000@gmail.com>
Thu, 25 Aug 2011 00:14:03 +0000 (20:14 -0400)
committerMichael Niedermayer <michaelni@gmx.at>
Wed, 19 Oct 2011 21:28:04 +0000 (23:28 +0200)
commite662b263d9c500270a8f1dc7e1b81b51d5bdfd4e
tree2c35f9fd139c6da27151019d5e0d69057d696f6c
parente31c5ebe1146d98d17a5121312c5444432c81904
Fixes avpicture_layout to not write past buffer end.

avpicture_get_size() returns the size of buffer required for avpicture_layout.
For pseudo-paletted formats (gray8...) this size does not include the palette.
However, avpicture_layout doesn't know this and still writes the palette. Consequently,
avpicture_layout writes passed the length of the buffer. This fixes it
by fixing avpicture_layout so that it doesn't write the palette for these formats.

Signed-off-by: Matthew Einhorn <moiein2000@gmail.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
libavcodec/imgconvert.c