ffmpeg.git
7 years agoUpdate for 0.8.12 n0.8.12
Michael Niedermayer [Sat, 9 Jun 2012 19:09:06 +0000 (21:09 +0200)]
Update for 0.8.12

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
7 years agompc8: fix channel checks
Michael Niedermayer [Sun, 3 Jun 2012 15:40:30 +0000 (17:40 +0200)]
mpc8: fix channel checks

fix heap array overflow

Found-by: Piotr Bandurski <ami_stuff@o2.pl>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 44c10168cff41c200825448b77cb8feff0d316c9)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
7 years agoh263: disable loop filter with lowres
Michael Niedermayer [Sun, 3 Jun 2012 12:41:21 +0000 (14:41 +0200)]
h263: disable loop filter with lowres

Fixes ticket1212

Found-by: Piotr Bandurski <ami_stuff@o2.pl>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit cc229d4e83889d1298f1a0863b55feec6c5c339a)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
7 years agowmv1: check that the input buffer is large enough
Michael Niedermayer [Fri, 1 Jun 2012 19:42:29 +0000 (21:42 +0200)]
wmv1: check that the input buffer is large enough

Fixes null ptr deref
Fixes Ticket1367

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit f23a2418fb0ccc56fdae4dbf83a5994cc917c475)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
7 years agoyopdec: check frame oddness to be within supported limits
Michael Niedermayer [Fri, 1 Jun 2012 13:52:20 +0000 (15:52 +0200)]
yopdec: check frame oddness to be within supported limits

Fixes Ticket1365

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit febc013dc5d6db1535a4f91cf02fa8089038937c)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
7 years agoyopdec: check that palette fits in the packet
Michael Niedermayer [Fri, 1 Jun 2012 13:51:50 +0000 (15:51 +0200)]
yopdec: check that palette fits in the packet

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit b6fdf8dea7aaf3cb9a979dce91f752c2ce3086a3)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
7 years ago8svx: fix crash
Michael Niedermayer [Thu, 31 May 2012 23:33:00 +0000 (01:33 +0200)]
8svx: fix crash

Fixes Ticket1377

Found-by: Piotr Bandurski <ami_stuff@o2.pl>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 03ce421c1361e4ce79468de8269ad51ba2ae4c16)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
7 years agobinkaudio: check number of channels
Paul B Mahol [Thu, 31 May 2012 08:58:31 +0000 (08:58 +0000)]
binkaudio: check number of channels

Fixes #1380.

Signed-off-by: Paul B Mahol <onemda@gmail.com>
(cherry picked from commit 824a6975ee066e944b7a20d1e220fd8974fb6174)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
7 years agoindeo5: check quant_mat
Michael Niedermayer [Thu, 31 May 2012 03:01:28 +0000 (05:01 +0200)]
indeo5: check quant_mat

prevents out of array read

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 8aaa00c3012d425ce50efffadb813ad62d1ff3d5)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
7 years agotruemotion1: Check index, fix out of array read
Michael Niedermayer [Wed, 30 May 2012 14:19:36 +0000 (16:19 +0200)]
truemotion1: Check index, fix out of array read

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit fd4c1c0b70b5a06dd572d7e27799a2f4c3d9b984)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
7 years agoiff: check if there is extradata
Paul B Mahol [Wed, 30 May 2012 07:50:32 +0000 (07:50 +0000)]
iff: check if there is extradata

Fixes #1368.

Signed-off-by: Paul B Mahol <onemda@gmail.com>
(cherry picked from commit 8f61526978697e51d3b9e61ea84daf13c42717af)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
7 years agoape: Fix null ptr dereference with files missing a seekatable.
Michael Niedermayer [Tue, 29 May 2012 17:50:15 +0000 (19:50 +0200)]
ape: Fix null ptr dereference with files missing a seekatable.

Such files are currently not supported as the table is used at several points

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit e7cb161515fc9fb6d30d1681d64d9ba7ad737a4e)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
7 years ago4xm: fix division by zero caused by bps<8
Michael Niedermayer [Tue, 29 May 2012 17:16:22 +0000 (19:16 +0200)]
4xm: fix division by zero caused by bps<8

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 1b8741a6843f3f4667c81c2d63d3182858aa534f)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
7 years agojvdec: check videosize
Michael Niedermayer [Mon, 28 May 2012 15:21:29 +0000 (17:21 +0200)]
jvdec: check videosize

Fixes null ptr dereference
fixes Ticket1364

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit b4904e804d3b1c56ac4f5d3386b15daae98fca2d)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
7 years agomotionpixels: check extradata size
Michael Niedermayer [Mon, 28 May 2012 15:17:49 +0000 (17:17 +0200)]
motionpixels: check extradata size

Fixes null ptr derefernce
Fixes Ticket1363

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 50122084a6b3be06781a2b3d8ec036f2d67c32e3)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
7 years agoiff_ilbm: fix null ptr deref
Michael Niedermayer [Mon, 28 May 2012 15:13:10 +0000 (17:13 +0200)]
iff_ilbm: fix null ptr deref

Fixes Ticket1362

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 849d4b041351ef8d77c4231cf417f997e79f9ab7)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
7 years agoyop: check for missing extradata
Michael Niedermayer [Mon, 28 May 2012 15:08:06 +0000 (17:08 +0200)]
yop: check for missing extradata

Fixes null ptr deref
Fixes Ticket1361

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 77a4c8b959fa9bc6bcaa42b40a0b046cdf3fec38)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
7 years agoxan: fix out of array read
Michael Niedermayer [Mon, 28 May 2012 15:04:38 +0000 (17:04 +0200)]
xan: fix out of array read

Fixes ticket1360

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 01900fcc45e99ee4556e0a5d87ff57b2f150dad4)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
7 years agocdgraphics: Fix out of array write
Michael Niedermayer [Mon, 28 May 2012 14:50:15 +0000 (16:50 +0200)]
cdgraphics: Fix out of array write

Fixes Ticket1359

Found-by: Piotr Bandurski <ami_stuff@o2.pl>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 1e5c7376c4ed733910845c9a09e272ac7696b1f4)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
7 years agoMerge remote-tracking branch 'qatar/release/0.7' into release/0.8
Michael Niedermayer [Mon, 4 Jun 2012 11:05:25 +0000 (13:05 +0200)]
Merge remote-tracking branch 'qatar/release/0.7' into release/0.8

* qatar/release/0.7:
  Update RELEASE file for 0.7.6
  Update changelog for 0.7.6 release
  ea: check chunk_size for validity.
  png: check bit depth for PAL8/Y400A pixel formats.
  x86: fix build with gcc 4.7
  qdm2: clip array indices returned by qdm2_get_vlc().
  kmvc: Check palsize.
  aacsbr: prevent out of bounds memcpy().
  rtpdec_asf: Fix integer underflow that could allow remote code execution
  dpcm: ignore extra unpaired bytes in stereo streams.
  tqi: Pass errors from the MB decoder
  h264: Add check for invalid chroma_format_idc
  adpcm: ADPCM Electronic Arts has always two channels
  h263dec: Disallow width/height changing with frame threads.
  vqavideo: return error if image size is not a multiple of block size
  celp filters: Do not read earlier than the start of the 'out' vector.
  motionpixels: Clip YUV values after applying a gradient.
  h263: more strictly forbid frame size changes with frame-mt.
  h264: additional protection against unsupported size/bitdepth changes.

Conflicts:
Changelog
RELEASE
libavcodec/aacsbr.c
libavcodec/h264_ps.c
libavcodec/pngdec.c
libavformat/rtpdec_asf.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>
7 years agoUpdate RELEASE file for 0.7.6
Reinhard Tartler [Sun, 3 Jun 2012 17:22:20 +0000 (19:22 +0200)]
Update RELEASE file for 0.7.6

7 years agoUpdate changelog for 0.7.6 release
Reinhard Tartler [Sun, 3 Jun 2012 17:22:09 +0000 (19:22 +0200)]
Update changelog for 0.7.6 release

7 years agoea: check chunk_size for validity.
Ronald S. Bultje [Fri, 4 May 2012 23:06:26 +0000 (16:06 -0700)]
ea: check chunk_size for validity.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 273e6af47b38391f2bcc157cca0423fe7fcbf55c)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit 6a86b705e1d4b72f0dddfbe23ad3eed9947001d5)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agopng: check bit depth for PAL8/Y400A pixel formats.
Ronald S. Bultje [Wed, 2 May 2012 17:58:55 +0000 (10:58 -0700)]
png: check bit depth for PAL8/Y400A pixel formats.

Wrong bit depth can lead to invalid rowsize values, which crashes the
decoder further down.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit d2205d6543881f2e6fa18c8a354bbcf91a1235f7)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit b8d6ba9d50e80fdce2ed74cdaffd4960df8a21c5)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agox86: fix build with gcc 4.7
Mans Rullgard [Tue, 26 Jul 2011 11:23:43 +0000 (12:23 +0100)]
x86: fix build with gcc 4.7

The upcoming gcc 4.7 has more advanced constant propagation
resulting some inline asm operands becoming constants and thus
emitted as literals, sometimes in contexts where this results
in invalid instructions.

This patch changes the constraints of the relevant operands
to "rm" thus forcing a valid type.  While obviously suboptimal,
this is what older gcc versions already did, and there is no
change to the code generated with these.

Signed-off-by: Mans Rullgard <mans@mansr.com>
(cherry picked from commit da4c7cce2100a4e4f9276b4f17e260be47b53f41)
Signed-off-by: Derek Buitenhuis <derek.buitenhuis@gmail.com>
7 years agoqdm2: clip array indices returned by qdm2_get_vlc().
Ronald S. Bultje [Wed, 2 May 2012 16:12:46 +0000 (16:12 +0000)]
qdm2: clip array indices returned by qdm2_get_vlc().

Prevents subsequent overreads when these numbers are used as indices
in arrays.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Justin Ruggles <justin.ruggles@gmail.com>
(cherry picked from commit 64953f67f98da2e787aeb45cc7f504390fa32a69)
Signed-off-by: Derek Buitenhuis <derek.buitenhuis@gmail.com>
Conflicts:

libavcodec/qdm2.c

7 years agokmvc: Check palsize.
Alex Converse [Thu, 26 Jan 2012 16:30:49 +0000 (17:30 +0100)]
kmvc: Check palsize.

Fixes: CVE-2011-3952

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Based on fix by Michael Niedermayer
(cherry picked from commit 386741f887714d3e46c9e8fe577e326a7964037b)
(cherry picked from commit 416849f2e06227b1b4a451c392f100db1d709a0c)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoaacsbr: prevent out of bounds memcpy().
Alex Converse [Tue, 10 Jan 2012 21:07:09 +0000 (13:07 -0800)]
aacsbr: prevent out of bounds memcpy().

Fixes Libav Bug 195.
Fixes CVE-2012-0850

This doesn't make the code handle sample rate or upsample/downsample
change properly but this is still a good sanity check.

Based on change by Michael Niedermayer.

Signed-off-by: Alex Converse <alex.converse@gmail.com>
(cherry picked from commit 17ce52912f59a74ecc265e062578fb1181456e18)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agortpdec_asf: Fix integer underflow that could allow remote code execution
Michael Niedermayer [Wed, 7 Sep 2011 12:12:42 +0000 (14:12 +0200)]
rtpdec_asf: Fix integer underflow that could allow remote code execution

Fixes MSVR-11-0088
Fixes CVE-2011-4031
Credit:  Jeong Wook Oh of Microsoft and Microsoft Vulnerability Research (MSVR)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit 5ea091fb5a12dc0210b8efdf30b573b87e21652b)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agodpcm: ignore extra unpaired bytes in stereo streams.
Alex Converse [Fri, 17 Feb 2012 22:13:40 +0000 (14:13 -0800)]
dpcm: ignore extra unpaired bytes in stereo streams.

Fixes: CVE-2011-3951

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
(cherry picked from commit ce7aee9b733134649a6ce2fa743e51733f33e67e)
(cherry picked from commit eaeaeb265fe46e1d81452960de918227541873b4)

Conflicts:

libavcodec/dpcm.c

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agotqi: Pass errors from the MB decoder
Michael Niedermayer [Mon, 19 Dec 2011 03:13:37 +0000 (04:13 +0100)]
tqi: Pass errors from the MB decoder

This silences some valgrind warnings.
CC: libav-stable@libav.org
Fixes second half of http://ffmpeg.org/trac/ffmpeg/ticket/794
Bug found by: Oana Stratulat

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit f85334f58e1286287d0547a49fa9c93b40cbf48f)
(cherry picked from commit 90290a5150e84fb138ccde57657dc03830f08c1c)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit 5872580e65aab026b77754eb184f97ba7cc6ea35)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoh264: Add check for invalid chroma_format_idc
Alexander Strange [Sat, 24 Mar 2012 21:32:14 +0000 (17:32 -0400)]
h264: Add check for invalid chroma_format_idc

Fixes a crash when FF_DEBUG_PICT_INFO is used.

Signed-off-by: Ronald S. Bultje <rsbultje@gmail.com>
(cherry picked from commit 6ef4063957aa5025c8d2cd757b6a537e4b6874df)

Fixes: CVE-2012-0851

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit 47132345184dc3d0ff962a57a1225564fe979548)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoadpcm: ADPCM Electronic Arts has always two channels
Janne Grunau [Thu, 5 Jan 2012 19:50:55 +0000 (20:50 +0100)]
adpcm: ADPCM Electronic Arts has always two channels

Fixes half of http://ffmpeg.org/trac/ffmpeg/ticket/794
Adresses CVE-2012-0852

(cherry picked from commit bb5b3940b08d8dad5b7e948e8f3b02cd2eb70716)

Conflicts:

libavcodec/adpcm.c

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoh263dec: Disallow width/height changing with frame threads.
Michael Niedermayer [Fri, 17 Feb 2012 21:35:10 +0000 (13:35 -0800)]
h263dec: Disallow width/height changing with frame threads.

Fixes CVE-2011-3937

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 71db86d53b5c6872cea31bf714a1a38ec78feaba)

Conflicts:

libavcodec/h263dec.c

Signed-off-by: Alex Converse <alex.converse@gmail.com>
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit 4be63587e110c05cda3101abf2e3745d919f3fae)

Conflicts:

libavcodec/h263dec.c

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agovqavideo: return error if image size is not a multiple of block size
Mans Rullgard [Mon, 23 Apr 2012 12:16:33 +0000 (13:16 +0100)]
vqavideo: return error if image size is not a multiple of block size

The decoder assumes in various places that the image size
is a multiple of the block size, and there is no obvious
way to support odd sizes.  Bailing out early if the header
specifies a bad size avoids various errors later on.

Fixes CVE-2012-0947.

Signed-off-by: Mans Rullgard <mans@mansr.com>
(cherry picked from commit 58b2e0f0f2fc96c1158e04f8aba95cbe6157a1a3)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit d5207e2af81580dd5e6277b354c8b459c3624f26)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agocelp filters: Do not read earlier than the start of the 'out' vector.
Alex Converse [Fri, 4 May 2012 17:27:03 +0000 (10:27 -0700)]
celp filters: Do not read earlier than the start of the 'out' vector.

CC: libav-stable@libav.org
(cherry picked from commit 37ddd3833219fa7b913fff3f5cccc6878b047e6b)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit 9ea94c44b1b414ab3bc6e9220ebb77621423ca38)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agomotionpixels: Clip YUV values after applying a gradient.
Alex Converse [Wed, 2 May 2012 19:08:03 +0000 (12:08 -0700)]
motionpixels: Clip YUV values after applying a gradient.

Prevents illegal reads on truncated and malformed input.

CC: libav-stable@libav.org
(cherry picked from commit b5da848facd41169283d7bfe568b83bdfa7fc42e)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit aaa6a666774eb02c351c84e80622a5c69e9b642e)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoh263: more strictly forbid frame size changes with frame-mt.
Ronald S. Bultje [Thu, 29 Mar 2012 19:24:10 +0000 (12:24 -0700)]
h263: more strictly forbid frame size changes with frame-mt.

Prevents crashes because the old check was incomplete.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 2d22d4307dcc1461f39a2ffb9c8db6c6b23fd080)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit 7fe4c8cb761b0fc8685dacf9f187311b9d124a52)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoh264: additional protection against unsupported size/bitdepth changes.
Ronald S. Bultje [Thu, 29 Mar 2012 23:37:09 +0000 (16:37 -0700)]
h264: additional protection against unsupported size/bitdepth changes.

Fixes crashes in codepaths not covered by original checks.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
(cherry picked from commit 732f9fcfe54fc9a0a7bbce53fe86b38744c2d301)

Conflicts:

libavcodec/h264.c

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit 746f1594d71dece6fd6f786447e19be9c200a07d)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoUpdate for 0.8.11 n0.8.11
Michael Niedermayer [Mon, 9 Apr 2012 16:49:50 +0000 (18:49 +0200)]
Update for 0.8.11

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
7 years agoChangelog, delete, its too inaccurate, git log is better.
Michael Niedermayer [Mon, 9 Apr 2012 15:53:17 +0000 (17:53 +0200)]
Changelog, delete, its too inaccurate, git log is better.

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
7 years agopngenc: Fix incorrect mask used for interlaced mode.
Michael Niedermayer [Tue, 20 Mar 2012 19:39:32 +0000 (20:39 +0100)]
pngenc: Fix incorrect mask used for interlaced mode.

Fixes Ticket1109

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 15db6a959057b92245a384909ec7d413d5c16461)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
7 years agodsp: fix diff_bytes_mmx() with small width
Michael Niedermayer [Sat, 17 Mar 2012 19:45:45 +0000 (20:45 +0100)]
dsp: fix diff_bytes_mmx() with small width

Fixes Ticket1068

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 73089eccd3e48539555349b36d8aabbf1cea416e)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
7 years agoMerge remote-tracking branch 'qatar/release/0.7' into release/0.8
Michael Niedermayer [Sun, 8 Apr 2012 19:08:39 +0000 (21:08 +0200)]
Merge remote-tracking branch 'qatar/release/0.7' into release/0.8

* qatar/release/0.7:
  Update changelog for 0.7.5 release

Merged-by: Michael Niedermayer <michaelni@gmx.at>
7 years agoReplace SSE2 instruction in scalarproduct_float_sse() by SSE equivalent.
ami_stuff [Thu, 22 Mar 2012 18:28:52 +0000 (19:28 +0100)]
Replace SSE2 instruction in scalarproduct_float_sse() by SSE equivalent.

Fixes an AAC decoding issue with the sample from ticket #213 on machines
with SSE but without SSE2.
Based on 89411a by Reimar.

(cherry picked from commit f6b78638086beae9bcab672d4c9de1790be5a928)

7 years agoMerge remote-tracking branch 'qatar/release/0.7' into release/0.8
Michael Niedermayer [Sun, 1 Apr 2012 23:25:31 +0000 (01:25 +0200)]
Merge remote-tracking branch 'qatar/release/0.7' into release/0.8

* qatar/release/0.7: (84 commits)
  id3v2: fix skipping extended header in id3v2.4
  Update RELEASE file for 0.7.5
  lcl: use AVERROR_INVALIDDATA instead of AVERROR_UNKNOWN
  kgv1dec: Increase offsets array size so it is large enough.
  kgv1: use avctx->get/release_buffer().
  kvmc: fix invalid reads
  nsvdec: Propagate error values instead of returning 0 in nsv_read_header().
  mjpegbdec: Fix overflow in SOS.
  shorten: Use separate pointers for the allocated memory for decoded samples.
  shorten: check for realloc failure (cherry picked from commit 9e5e2c2d010c05c10337e9c1ec9d0d61495e0c9c)
  atrac3: Fix crash in tonal component decoding.
  ws_snd1: Fix wrong samples count and crash.
  ws_snd: add some checks to prevent buffer overread or overwrite. (cherry picked from commit 417364ce1f979031ef6fee661fc15e1869bdb1b4)
  ws_snd: decode to AV_SAMPLE_FMT_U8 instead of S16.
  dca: include libavutil/mathematics.h for possibly missing M_SQRT1_2
  h264: stricter reference limit enforcement.
  jvdec: unbreak video decoding
  xxan: don't read before start of buffer in av_memcpy_backptr().
  dsicinvideo: validate buffer offset before copying pixels.
  huffyuv: add padding to classic (v1) huffman tables.
  ...

Conflicts:
RELEASE
libavcodec/atrac3.c
libavcodec/h264.c
libavcodec/h264_parser.c
libavcodec/kgv1dec.c
libavcodec/shorten.c
libavcodec/svq3.c
libavcodec/ws-snd1.c
libavcodec/xxan.c
libswscale/utils.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>
7 years agoUpdate changelog for 0.7.5 release
Reinhard Tartler [Sun, 1 Apr 2012 17:45:27 +0000 (19:45 +0200)]
Update changelog for 0.7.5 release

7 years agoid3v2: fix skipping extended header in id3v2.4
Anton Khirnov [Sat, 31 Mar 2012 05:52:42 +0000 (07:52 +0200)]
id3v2: fix skipping extended header in id3v2.4

In v2.4, the length includes the length field itself.
(cherry picked from commit ddb4431208745ea270dce8fce4cba999f0ed4303)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
7 years agoUpdate RELEASE file for 0.7.5
Reinhard Tartler [Sun, 1 Apr 2012 17:08:06 +0000 (19:08 +0200)]
Update RELEASE file for 0.7.5

7 years agolcl: use AVERROR_INVALIDDATA instead of AVERROR_UNKNOWN
Reinhard Tartler [Sun, 18 Mar 2012 18:08:15 +0000 (19:08 +0100)]
lcl: use AVERROR_INVALIDDATA instead of AVERROR_UNKNOWN

While bogus, this change avoids the necessity to backport
AVERROR_UNKNOWN, which is not entirely trivial.

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agokgv1dec: Increase offsets array size so it is large enough.
Michael Niedermayer [Wed, 25 Jan 2012 22:23:35 +0000 (23:23 +0100)]
kgv1dec: Increase offsets array size so it is large enough.

Fixes CVE-2011-3945

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 807a045ab7f51993a2c1b3116016cbbd4f3d20d6)

Signed-off-by: Alex Converse <alex.converse@gmail.com>
(cherry picked from commit a02e8df973f5478ec82f4c507f5b5b191a5ecb6b)
(cherry picked from commit d5f2382d0389ed47a566ea536887af908bf9b14f)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agokgv1: use avctx->get/release_buffer().
Ronald S. Bultje [Thu, 29 Dec 2011 17:07:32 +0000 (09:07 -0800)]
kgv1: use avctx->get/release_buffer().

Also fixes crashes on corrupt bitstreams.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 33cd32b389864f2437c94e6fd7dc109ff5f0ed06)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit e537dc230b2e123be8aebdaeee5a7d7787328b0b)

Conflicts:

libavcodec/kgv1dec.c
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agokvmc: fix invalid reads
Gaurav Narula [Mon, 12 Dec 2011 14:54:54 +0000 (20:24 +0530)]
kvmc: fix invalid reads

Signed-off-by: Janne Grunau <janne-libav@jannau.net>
(cherry picked from commit ad3161ec1d70291efcf40121d703ef73c0b08e5b)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agonsvdec: Propagate error values instead of returning 0 in nsv_read_header().
Diego Biurrun [Thu, 23 Jun 2011 11:27:21 +0000 (13:27 +0200)]
nsvdec: Propagate error values instead of returning 0 in nsv_read_header().

This eliminates a warning about a set-but-unused variable.
(cherry picked from commit 35fa0d47585cef28cd8191dccf0607d90c7667a6)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agomjpegbdec: Fix overflow in SOS.
Alex Converse [Wed, 25 Jan 2012 21:39:24 +0000 (13:39 -0800)]
mjpegbdec: Fix overflow in SOS.

Based in part by a fix from Michael Niedermayer <michaelni@gmx.at>

Fixes CVE-2011-3947

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
(cherry picked from commit b57d262412204e54a7ef8fa1b23ff4dcede622e5)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit 083a8a00373b12dc06b8ae4c49eec61fb5e55f4b)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoshorten: Use separate pointers for the allocated memory for decoded samples.
Michael Niedermayer [Sun, 25 Dec 2011 11:28:50 +0000 (12:28 +0100)]
shorten: Use separate pointers for the allocated memory for decoded samples.

Fixes invalid free() if any of the buffers are not allocated due to either
not decoding a header or an error prior to allocating all buffers.

Fixes CVE-2012-0858
CC: libav-stable@libav.org
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Signed-off-by: Justin Ruggles <justin.ruggles@gmail.com>
(cherry picked from commit 204cb29b3c84a74cbcd059d353c70c8bdc567d98)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit 6fc3287b9ccece290c5881b92948772bbf72e68c)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoshorten: check for realloc failure (cherry picked from commit 9e5e2c2d010c05c10337e9c...
Justin Ruggles [Thu, 15 Sep 2011 22:08:52 +0000 (18:08 -0400)]
shorten: check for realloc failure (cherry picked from commit 9e5e2c2d010c05c10337e9c1ec9d0d61495e0c9c)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoatrac3: Fix crash in tonal component decoding.
Michael Niedermayer [Sat, 17 Dec 2011 02:18:58 +0000 (03:18 +0100)]
atrac3: Fix crash in tonal component decoding.

Add a check to avoid writing past the end of the channel_unit.components[]
array.

Bug Found by: cosminamironesei
Fixes CVE-2012-0853
CC: libav-stable@libav.org
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Signed-off-by: Justin Ruggles <justin.ruggles@gmail.com>
(cherry picked from commit c509f4f74713b035a06f79cb4d00e708f5226bc5)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit f43b6e2b1ed47a1254a5d44c700a7fad5e9784be)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agows_snd1: Fix wrong samples count and crash.
Michael Niedermayer [Sat, 24 Dec 2011 23:10:27 +0000 (00:10 +0100)]
ws_snd1: Fix wrong samples count and crash.

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 9fb7a5af97d8c084c3af2566070d09eae0ab49fc)

Addresses CVE-2012-0848

Reviewed-by: Justin Ruggles <justin.ruggles@gmail.com>
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit 697a45d861b7cd6a96718383a44f41348487f844)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agows_snd: add some checks to prevent buffer overread or overwrite. (cherry picked from...
Justin Ruggles [Mon, 12 Sep 2011 13:41:06 +0000 (09:41 -0400)]
ws_snd: add some checks to prevent buffer overread or overwrite. (cherry picked from commit 417364ce1f979031ef6fee661fc15e1869bdb1b4)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agows_snd: decode to AV_SAMPLE_FMT_U8 instead of S16.
Justin Ruggles [Mon, 12 Sep 2011 12:55:43 +0000 (08:55 -0400)]
ws_snd: decode to AV_SAMPLE_FMT_U8 instead of S16.

8-bit unsigned is the native sample format.
(cherry picked from commit 2322ced8da990835717a176b8d2c32961cfecd3e)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agodca: include libavutil/mathematics.h for possibly missing M_SQRT1_2
Kostya Shishkov [Wed, 7 Mar 2012 19:07:17 +0000 (20:07 +0100)]
dca: include libavutil/mathematics.h for possibly missing M_SQRT1_2

Signed-off-by: Janne Grunau <janne-libav@jannau.net>
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoh264: stricter reference limit enforcement.
Ronald S. Bultje [Tue, 13 Mar 2012 23:26:44 +0000 (16:26 -0700)]
h264: stricter reference limit enforcement.

Progressive images can have only 16 references, error out if there are
more, since the data is almost certainly corrupt, and the invalid value
will lead to random crashes or invalid writes later on.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit e0febda22d0e0fab094a9c886b0e0f0f662df1ef)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agojvdec: unbreak video decoding
Paul B Mahol [Wed, 14 Mar 2012 03:02:02 +0000 (03:02 +0000)]
jvdec: unbreak video decoding

The safe bitstream reader broke it since the buffer size was specified
in bytes instead of bits.

Signed-off-by: Janne Grunau <janne-libav@jannau.net>
CC: libav-stable@libav.org
(cherry picked from commit a1c036e961a32f7208e7315dabfa0ee99d779edb)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoxxan: don't read before start of buffer in av_memcpy_backptr().
Ronald S. Bultje [Fri, 9 Mar 2012 00:32:46 +0000 (16:32 -0800)]
xxan: don't read before start of buffer in av_memcpy_backptr().

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit f1279e286b00e99f343adb51e251f036a3df6f32)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agodsicinvideo: validate buffer offset before copying pixels.
Ronald S. Bultje [Sun, 11 Mar 2012 14:28:54 +0000 (07:28 -0700)]
dsicinvideo: validate buffer offset before copying pixels.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit c95fefa0420be9cc0f09a95041acf11114aaacd0)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agohuffyuv: add padding to classic (v1) huffman tables.
Ronald S. Bultje [Thu, 8 Mar 2012 00:29:23 +0000 (16:29 -0800)]
huffyuv: add padding to classic (v1) huffman tables.

We slightly overread the input buffer, so we require
padding at the end of the buffer, as is documented in the
get_bits API. Without padding, we'll read uninitialized
data or beyond the end of the .rodata, which may crash.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 4ffe5e2aa5241f8da9afd2c8fbc854dcc916c5f9)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agotiffdec: Prevent illegal memory access caused by recycled pointers.
Alex Converse [Wed, 7 Mar 2012 01:00:29 +0000 (17:00 -0800)]
tiffdec: Prevent illegal memory access caused by recycled pointers.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit fd0be63049ed46660993d0550a4f0847a0b942ea)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agowma: fix off-by-one in array bounds check.
Ronald S. Bultje [Wed, 7 Mar 2012 22:18:14 +0000 (14:18 -0800)]
wma: fix off-by-one in array bounds check.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit b4bccf3e4e58f6fe58043791ca09db01a4343fac)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoraw: move buffer size check up.
Ronald S. Bultje [Wed, 7 Mar 2012 00:08:10 +0000 (16:08 -0800)]
raw: move buffer size check up.

This way, it protects against overreads for 4bpp/2bpp content also.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit cc5dd632cecc5114717d0b90f8c2be162b1c6ee8)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agosmacker: error out if palette copy-with-offset overruns palette size.
Ronald S. Bultje [Wed, 7 Mar 2012 01:24:20 +0000 (17:24 -0800)]
smacker: error out if palette copy-with-offset overruns palette size.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit a93b572ae4f517ce0c35cf085167c318e9215908)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agosvq3: protect against negative quantizers.
Ronald S. Bultje [Tue, 6 Mar 2012 01:03:32 +0000 (17:03 -0800)]
svq3: protect against negative quantizers.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 11b940a1a8e7e5d5b212935a3ce78aeda577f5f2)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agomov: Add more HDV and XDCAM FourCCs.
Alex Converse [Tue, 21 Feb 2012 23:37:35 +0000 (15:37 -0800)]
mov: Add more HDV and XDCAM FourCCs.

Reference: VLC
(cherry picked from commit b142496c5630b9bc88fb9eaccae7f6bd62fb23e7)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agomov: Add support for MPEG2 HDV 720p24 (hdv4)
Alex Converse [Tue, 21 Feb 2012 22:08:02 +0000 (14:08 -0800)]
mov: Add support for MPEG2 HDV 720p24 (hdv4)

(cherry picked from commit 0ad522afb3a3b3d22402ecb82dd4609f7655031b)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoh263dec: Disallow width/height changing with frame threads.
Michael Niedermayer [Fri, 17 Feb 2012 21:35:10 +0000 (13:35 -0800)]
h263dec: Disallow width/height changing with frame threads.

Fixes CVE-2011-3937

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 71db86d53b5c6872cea31bf714a1a38ec78feaba)

Conflicts:

libavcodec/h263dec.c

Signed-off-by: Alex Converse <alex.converse@gmail.com>
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agotiff: Make the TIFF_LONG and TIFF_SHORT types unsigned.
Alex Converse [Thu, 23 Feb 2012 18:22:51 +0000 (10:22 -0800)]
tiff: Make the TIFF_LONG and TIFF_SHORT types unsigned.

TIFF v6.0 (unimplemented) adds signed equivalents.
(cherry picked from commit e32548d1331ce05a054f1028fcdda8823a4f215a)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agosvq3: Prevent illegal reads while parsing extradata.
Alex Converse [Fri, 10 Feb 2012 04:21:47 +0000 (20:21 -0800)]
svq3: Prevent illegal reads while parsing extradata.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
(cherry picked from commit 9e1db721c4329f4ac166a0bcc002c8d75f831aba)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agodv: Fix small overread in audio frequency table.
Alex Converse [Fri, 10 Feb 2012 01:11:55 +0000 (17:11 -0800)]
dv: Fix small overread in audio frequency table.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
(cherry picked from commit 0ab3687924457cb4fd81897bd39ab3cc5b699588)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoac3: Do not read past the end of ff_ac3_band_start_tab.
Mans Rullgard [Tue, 31 Jan 2012 18:20:33 +0000 (10:20 -0800)]
ac3: Do not read past the end of ff_ac3_band_start_tab.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Alex Converse <alex.converse@gmail.com>
(cherry picked from commit 034b03e7a0e8e4f8f66c82b736f2c0aa7c063ec0)
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agodv: Fix small stack overread related to CVE-2011-3929 and CVE-2011-3936.
Alex Converse [Thu, 26 Jan 2012 23:08:26 +0000 (15:08 -0800)]
dv: Fix small stack overread related to CVE-2011-3929 and CVE-2011-3936.

Found with asan.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Alex Converse <alex.converse@gmail.com>
(cherry picked from commit 2d1c0dea5f6b91bec7f5fa53ec050913d851e366)
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agodv: Fix null pointer dereference due to ach=0
Michael Niedermayer [Tue, 24 Jan 2012 16:51:40 +0000 (17:51 +0100)]
dv: Fix null pointer dereference due to ach=0

dv: Fix null pointer dereference due to ach=0

Fixes part2 of CVE-2011-3929

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Reviewed-by: Roman Shaposhnik <roman@shaposhnik.org>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Signed-off-by: Alex Converse <alex.converse@gmail.com>
(cherry picked from commit 5a396bb3a66a61a68b80f2369d0249729bf85e04)
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agodv: check stype
Michael Niedermayer [Tue, 24 Jan 2012 16:48:23 +0000 (17:48 +0100)]
dv: check stype

dv: check stype

Fixes part1 of CVE-2011-3929
Possibly fixes part of CVE-2011-3936

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Reviewed-by: Roman Shaposhnik <roman@shaposhnik.org>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Signed-off-by: Alex Converse <alex.converse@gmail.com>
(cherry picked from commit 635bcfccd439480003b74a665b5aa7c872c1ad6b)
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agonsvdec: Propagate errors
Alex Converse [Fri, 27 Jan 2012 01:23:09 +0000 (17:23 -0800)]
nsvdec: Propagate errors

Related to CVE-2011-3940.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
(cherry picked from commit c898431ca5ef2a997fe9388b650f658fb60783e5)

Conflicts:

libavformat/nsvdec.c

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agonsvdec: Be more careful with av_malloc().
Alex Converse [Fri, 27 Jan 2012 01:21:46 +0000 (17:21 -0800)]
nsvdec: Be more careful with av_malloc().

Check results for av_malloc() and fix an overflow in one call.

Related to CVE-2011-3940.

Based in part on work from Michael Niedermayer.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
(cherry picked from commit 8fd8a48263ff1437f9d02d7e78dc63efb9b5ed3a)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agonsvdec: Fix use of uninitialized streams.
Michael Niedermayer [Tue, 24 Jan 2012 21:20:26 +0000 (22:20 +0100)]
nsvdec: Fix use of uninitialized streams.

Fixes CVE-2011-3940 (Out of bounds read resulting in out of bounds write)

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 5c011706bc752d34bc6ada31d7df2ca0c9af7c6b)

Signed-off-by: Alex Converse <alex.converse@gmail.com>
(cherry picked from commit 6a89b41d9780325ba6d89a37f2aeb925aa68e6a3)
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoFix format string vulnerability detected by -Wformat-security.
Fabian Greffrath [Mon, 5 Mar 2012 15:06:01 +0000 (16:06 +0100)]
Fix format string vulnerability detected by -Wformat-security.

Signed-off-by: Diego Biurrun <diego@biurrun.de>
(cherry picked from commit c9dbac36ad4bac07f6c1d06d465e361ab55bcb95)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoh264: fix mmxext chroma deblock to use correct TC values. (cherry picked from commit...
Ronald S. Bultje [Sun, 26 Feb 2012 16:57:14 +0000 (08:57 -0800)]
h264: fix mmxext chroma deblock to use correct TC values. (cherry picked from commit b0c4f04338234ee011d7b704621347ef232294fe)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agocscd: use negative error values to indicate decode_init() failures.
Ronald S. Bultje [Wed, 29 Feb 2012 21:55:09 +0000 (13:55 -0800)]
cscd: use negative error values to indicate decode_init() failures.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 8a9faf33f2b4f40afbc3393b2be49867cea0c92d)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoh264: prevent overreads in intra PCM decoding.
Ronald S. Bultje [Wed, 29 Feb 2012 02:48:27 +0000 (18:48 -0800)]
h264: prevent overreads in intra PCM decoding.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit d1604b3de96575195b219028e2c4f08b2259aa7d)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agowmaenc: fix m/s stereo encoding for the first frame
Justin Ruggles [Fri, 2 Mar 2012 22:11:25 +0000 (17:11 -0500)]
wmaenc: fix m/s stereo encoding for the first frame

We need to set ms_stereo in encode_init() in order to avoid incorrectly
encoding the first frame as non-m/s while flagging it as m/s. Fixes an
uncomfortable pop in the left channel at the start of playback.

CC:libav-stable@libav.org
(cherry picked from commit 51ddf35c9017018e58c15275ff5b129647a0c94d)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agowmaenc: limit allowed sample rate to 48kHz
Justin Ruggles [Fri, 2 Mar 2012 21:27:57 +0000 (16:27 -0500)]
wmaenc: limit allowed sample rate to 48kHz

ff_wma_init() allows up to 50kHz, but this generates an exponent band
size table that requires 65 bands. The code assumes 25 bands in many
places, and using sample rates higher than 48kHz will lead to buffer
overwrites.

CC:libav-stable@libav.org
(cherry picked from commit 1ec075cfecac01f9a289965db06f76365b0b1737)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agowmaenc: limit block_align to MAX_CODED_SUPERFRAME_SIZE
Justin Ruggles [Fri, 2 Mar 2012 21:10:00 +0000 (16:10 -0500)]
wmaenc: limit block_align to MAX_CODED_SUPERFRAME_SIZE

This is near the theoretical limit for wma frame size and is the most that
our decoder can handle. Allowing higher bit rates will just end up padding
each frame with empty bytes.

Fixes invalid writes for avconv when using very high bit rates.

CC:libav-stable@libav.org
(cherry picked from commit c2b8dea1828f35c808adcf12615893d5c740bc0a)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agowmaenc: require a large enough output buffer to prevent overwrites
Justin Ruggles [Fri, 2 Mar 2012 21:33:33 +0000 (16:33 -0500)]
wmaenc: require a large enough output buffer to prevent overwrites

The maximum theoretical frame size is around 17000 bytes. Although in
practice it will generally be much smaller, we require a larger buffer
just to be safe.

CC: libav-stable@libav.org
(cherry picked from commit dfc4fdedf8cfc56a505579b1f2c1c5efbce4b97e)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agomatroska: check buffer size for RM-style byte reordering.
Ronald S. Bultje [Fri, 2 Mar 2012 01:01:22 +0000 (17:01 -0800)]
matroska: check buffer size for RM-style byte reordering.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 9c239f6026a170866a4a0c96908980ac2cfaa8b3)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agowmadec: Verify bitstream size makes sense before calling init_get_bits.
Alex Converse [Fri, 27 Jan 2012 22:24:07 +0000 (14:24 -0800)]
wmadec: Verify bitstream size makes sense before calling init_get_bits.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
(cherry picked from commit 48f1e5212c90b511c90fa0449655abb06a9edda2)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agorv10/20: Fix a buffer overread caused by losing track of the remaining buffer size.
Alex Converse [Thu, 1 Mar 2012 22:07:22 +0000 (14:07 -0800)]
rv10/20: Fix a buffer overread caused by losing track of the remaining buffer size.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 2f6528537fdd88820f3a4683d5e595d7b3a62689)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agolcl: return negative error codes on decode_init() errors.
Ronald S. Bultje [Thu, 1 Mar 2012 01:50:28 +0000 (17:50 -0800)]
lcl: return negative error codes on decode_init() errors.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit bd17a40a7e0eba21b5d27c67aff795e2910766e4)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agohuffyuv: do not abort on unknown pix_fmt; instead, return an error.
Ronald S. Bultje [Thu, 1 Mar 2012 17:41:22 +0000 (09:41 -0800)]
huffyuv: do not abort on unknown pix_fmt; instead, return an error.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 63c9de6469005974288f4e4d89fc79a590e38c06)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agovmnc: return error on decode_init() failure.
Ronald S. Bultje [Wed, 29 Feb 2012 03:00:48 +0000 (19:00 -0800)]
vmnc: return error on decode_init() failure.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 07a180972fb369bb59bf6d4f8edb4598c51e80d2)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agorpza: error out on buffer overreads.
Ronald S. Bultje [Wed, 29 Feb 2012 01:04:33 +0000 (17:04 -0800)]
rpza: error out on buffer overreads.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 78e9852a2e3b198ecd69ffa0deab3fa22a8e5378)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>