ffmpeg.git
7 years agoh263: more strictly forbid frame size changes with frame-mt.
Ronald S. Bultje [Thu, 29 Mar 2012 19:24:10 +0000 (12:24 -0700)]
h263: more strictly forbid frame size changes with frame-mt.

Prevents crashes because the old check was incomplete.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 2d22d4307dcc1461f39a2ffb9c8db6c6b23fd080)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoh264: additional protection against unsupported size/bitdepth changes.
Ronald S. Bultje [Thu, 29 Mar 2012 23:37:09 +0000 (16:37 -0700)]
h264: additional protection against unsupported size/bitdepth changes.

Fixes crashes in codepaths not covered by original checks.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
(cherry picked from commit 732f9fcfe54fc9a0a7bbce53fe86b38744c2d301)

Conflicts:

libavcodec/h264.c

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agotta: prevents overflows for 32bit integers in header.
Ronald S. Bultje [Thu, 29 Mar 2012 19:44:55 +0000 (12:44 -0700)]
tta: prevents overflows for 32bit integers in header.

This prevents sample_rate/data_length from going negative, which
caused various crashes and undefined behaviour further down.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit ac80b812cd177553339467ea12548d71c9ef6865)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agottadec: CRC checking
Paul B Mahol [Sat, 11 Feb 2012 21:30:30 +0000 (21:30 +0000)]
ttadec: CRC checking

Signed-off-by: Paul B Mahol <onemda@gmail.com>
Signed-off-by: Justin Ruggles <justin.ruggles@gmail.com>
(cherry picked from commit 2af3dc8698707f800f83f5fc890571a6a119866e)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agotta: use skip_bits_long()
Paul B Mahol [Sun, 5 Feb 2012 19:39:13 +0000 (19:39 +0000)]
tta: use skip_bits_long()

Signed-off-by: Paul B Mahol <onemda@gmail.com>
Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit 9aff2d17533576f4ff52531e534f1319fb36a590)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoapedec: check bits <= 32.
Michael Niedermayer [Thu, 29 Mar 2012 17:52:21 +0000 (17:52 +0000)]
apedec: check bits <= 32.

Fixes a floating-point exception further down.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Signed-off-by: Ronald S. Bultje <rsbultje@gmail.com>
Signed-off-by: Derek Buitenhuis <derek.buitenhuis@gmail.com>
(cherry picked from commit 420d1df2e2a857eae45fa947e16eae7494793d57)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agotruemotion: forbid invalid VLC bitsizes and token values.
Ronald S. Bultje [Thu, 29 Mar 2012 17:25:04 +0000 (10:25 -0700)]
truemotion: forbid invalid VLC bitsizes and token values.

SHOW_UBITS() is only defined up to n_bits is 25, therefore forbid
values larger than this in get_vlc2() (max_bits). tokens[][] can be
used as an index in deltas[], which has a size of 64, so ensure the
values are smaller than that.

This prevents crashes on corrupt bitstreams.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit b7b1509d06d3696d3b944791227fe198ded0654b)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agomov: don't overwrite existing indexes.
Ronald S. Bultje [Wed, 28 Mar 2012 19:56:07 +0000 (12:56 -0700)]
mov: don't overwrite existing indexes.

Prevents all kind of badness if files contain multiple
indexes.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 4f7c7624c0db185c48c59d95d745ab3f7851a5b4)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agotruemotion2: handle out-of-frame motion vectors through edge extension.
Ronald S. Bultje [Thu, 29 Mar 2012 16:29:03 +0000 (09:29 -0700)]
truemotion2: handle out-of-frame motion vectors through edge extension.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit bf39d3b59d85e5734babe48b61b8d92d18188185)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agolzw: prevent buffer overreads.
Ronald S. Bultje [Thu, 29 Mar 2012 00:06:00 +0000 (17:06 -0700)]
lzw: prevent buffer overreads.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit ddcf67c8a51c67b122a826d8b5819e96d591d813)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agotruemotion2: convert packet header reading to bytestream2.
Ronald S. Bultje [Wed, 28 Mar 2012 18:53:13 +0000 (11:53 -0700)]
truemotion2: convert packet header reading to bytestream2.

Also use correct buffer sizes in calls to tm2_read_stream(). Together,
this prevents overreads.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit bd508d435b94584db460c684e30ea7ce180cf50f)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agolagarith: fix buffer overreads.
Ronald S. Bultje [Tue, 27 Mar 2012 19:26:46 +0000 (12:26 -0700)]
lagarith: fix buffer overreads.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 0a82f5275f719e6e369a807720a2c3603aa0ddd9)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoraw: forward avpicture_fill() error code in raw_decode().
Ronald S. Bultje [Tue, 27 Mar 2012 01:02:08 +0000 (18:02 -0700)]
raw: forward avpicture_fill() error code in raw_decode().

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 98df2e24141cd00a557ef10ed7af2b956200cd80)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agovc1: Do not read from array if index is invalid.
Mashiat Sarker Shakkhar [Sat, 24 Mar 2012 22:49:34 +0000 (15:49 -0700)]
vc1: Do not read from array if index is invalid.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Ronald S. Bultje <rsbultje@gmail.com>
(cherry picked from commit 95b192de5d05f3e1542e7b2378cdefbc195f5185)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoutvideo: port header reading to bytestream2.
Ronald S. Bultje [Fri, 23 Mar 2012 00:25:22 +0000 (17:25 -0700)]
utvideo: port header reading to bytestream2.

Fixes crash during slice size reading if slice_end goes negative.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit ec0ed97b046d46421db72c4911d2bbe28bbe5741)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agobytestream: add more unchecked variants for bytestream2 API
Paul B Mahol [Tue, 13 Mar 2012 14:14:59 +0000 (14:14 +0000)]
bytestream: add more unchecked variants for bytestream2 API

Signed-off-by: Paul B Mahol <onemda@gmail.com>
Signed-off-by: Ronald S. Bultje <rsbultje@gmail.com>
(cherry picked from commit f1ce053cd0e0d7dc67fa61f32bcd8b6ee5e5c490)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agobytestream: K&R formatting cosmetics
Aneesh Dogra [Wed, 8 Feb 2012 18:07:20 +0000 (23:37 +0530)]
bytestream: K&R formatting cosmetics

Signed-off-by: Diego Biurrun <diego@biurrun.de>
(cherry picked from commit ab9ae401525d301a31ec695bf39103502db6afeb)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agobytestream: Add bytestream2 writing API.
Aneesh Dogra [Mon, 6 Feb 2012 20:09:22 +0000 (01:39 +0530)]
bytestream: Add bytestream2 writing API.

Signed-off-by: Justin Ruggles <justin.ruggles@gmail.com>
(cherry picked from commit db7d45237ab6fc7fe90ec861cb756b2a109504a4)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoaac: Reset PS parameters on header decode failure.
Alex Converse [Wed, 21 Mar 2012 17:11:02 +0000 (10:11 -0700)]
aac: Reset PS parameters on header decode failure.

If the next header frame codes zero envelopes the previous frame's
values will be used. Consequently the invalid values must be cleared.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit a237b38021cd3009cc78eeb974b596085f2fe393)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agomov: Do not read past the end of the ctts_data table.
Alex Converse [Wed, 21 Mar 2012 18:24:10 +0000 (11:24 -0700)]
mov: Do not read past the end of the ctts_data table.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 86f2ae06b92d42580ae7ebd86d52c9b7acbc2f13)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoxwma: Validate channels and bits_per_coded_sample.
Alex Converse [Wed, 21 Mar 2012 17:58:07 +0000 (10:58 -0700)]
xwma: Validate channels and bits_per_coded_sample.

This prevents a SIGFPE later on.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 5023b89bba198b2f8e43b7f555aeb9c30d33db9f)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoasf: reset side data elements on packet copy.
Ronald S. Bultje [Wed, 21 Mar 2012 23:10:37 +0000 (16:10 -0700)]
asf: reset side data elements on packet copy.

Prevents crash (double free) when free()ing the original packet.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit e73c6aaabff1169899184c382385fe9afae5b068)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agovqa: check palette chunk size before reading data.
Ronald S. Bultje [Wed, 21 Mar 2012 22:19:31 +0000 (15:19 -0700)]
vqa: check palette chunk size before reading data.

Prevents overreads beyond buffer boundaries.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 75d7975268394f4f16294b68ec6d6d5ac30da3ac)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agovqavideo: port to bytestream2 API
Paul B Mahol [Fri, 16 Mar 2012 00:56:41 +0000 (00:56 +0000)]
vqavideo: port to bytestream2 API

Protects against overreads.

Signed-off-by: Paul B Mahol <onemda@gmail.com>
Signed-off-by: Ronald S. Bultje <rsbultje@gmail.com>
(cherry picked from commit 5a3a906ba29b53fa34d3047af78d9f8fd7678256)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agowmavoice: fix stack overread.
Ronald S. Bultje [Wed, 21 Mar 2012 22:47:11 +0000 (15:47 -0700)]
wmavoice: fix stack overread.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 262196445cf03fda0f7e41c4b968f4f7bf060e6b)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoindeo4: fix out-of-bounds function call.
Ronald S. Bultje [Wed, 21 Mar 2012 17:39:10 +0000 (10:39 -0700)]
indeo4: fix out-of-bounds function call.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Kostya Shishkov <kostya.shishkov@gmail.com>
(cherry picked from commit 68fd077f68bdde864bb7328d72a040849c616261)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoRead preset files with suffix .avpreset
Reinhard Tartler [Sun, 18 Mar 2012 08:26:32 +0000 (09:26 +0100)]
Read preset files with suffix .avpreset

The preset files have been renamed some time ago.

CC: libav-stable@libav.org
(cherry picked from commit 050dc127787e91d8ee4b341046c74fe6e74e3285)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agomimic: don't use self as reference, and report completion at end of decode().
Ronald S. Bultje [Fri, 16 Mar 2012 21:04:00 +0000 (14:04 -0700)]
mimic: don't use self as reference, and report completion at end of decode().

Fixes hangs on corrupt samples that reference self-frames.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 80387f0e2568746dce4a68e2217297029a053dae)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agompeg4: report frame decoding completion at ff_MPV_frame_end().
Ronald S. Bultje [Fri, 16 Mar 2012 21:16:56 +0000 (14:16 -0700)]
mpeg4: report frame decoding completion at ff_MPV_frame_end().

Prevents hangs on corrupt input.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit c6ccb96bc955b2087ec71033d99b3dcd5203eaf2)

Conflicts:

libavcodec/mpegvideo.c

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoid3v2: fix skipping extended header in id3v2.4
Anton Khirnov [Sat, 31 Mar 2012 05:52:42 +0000 (07:52 +0200)]
id3v2: fix skipping extended header in id3v2.4

In v2.4, the length includes the length field itself.
(cherry picked from commit ddb4431208745ea270dce8fce4cba999f0ed4303)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
7 years agoUpdate Changelog for the 0.8.1 Release
Reinhard Tartler [Thu, 15 Mar 2012 07:57:33 +0000 (08:57 +0100)]
Update Changelog for the 0.8.1 Release

7 years agodca: include libavutil/mathematics.h for possibly missing M_SQRT1_2
Kostya Shishkov [Wed, 7 Mar 2012 19:07:17 +0000 (20:07 +0100)]
dca: include libavutil/mathematics.h for possibly missing M_SQRT1_2

Signed-off-by: Janne Grunau <janne-libav@jannau.net>
7 years agodca: don't use av_clip_uintp2().
Ronald S. Bultje [Wed, 7 Mar 2012 19:06:20 +0000 (11:06 -0800)]
dca: don't use av_clip_uintp2().

The argument is not a literal, thus causing the ARM v6 or later
builds to break.

Signed-off-by: Janne Grunau <janne-libav@jannau.net>
7 years agosnow: check reference frame indices.
Michael Niedermayer [Fri, 2 Mar 2012 19:53:00 +0000 (20:53 +0100)]
snow: check reference frame indices.

Fixes NULL ptr dereference

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Ronald S. Bultje <rsbultje@gmail.com>
(cherry picked from commit 1f8ff2b13cbfef790385818664ed12e763e7c75b)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agosnow: reject unsupported chroma shifts.
Michael Niedermayer [Fri, 9 Mar 2012 23:08:32 +0000 (00:08 +0100)]
snow: reject unsupported chroma shifts.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Ronald S. Bultje <rsbultje@gmail.com>
(cherry picked from commit c9837954e7b968d44f82e7cdb7618e9f523b196c)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoxa_adpcm: limit filter to prevent xa_adpcm_table[] array bounds overruns.
Ronald S. Bultje [Tue, 13 Mar 2012 19:28:35 +0000 (12:28 -0700)]
xa_adpcm: limit filter to prevent xa_adpcm_table[] array bounds overruns.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 86020073dbb9a3a9d1fbb76345b2ca29ba1f13d2)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoh264: increase reference poc list from 16 to 32.
Ronald S. Bultje [Tue, 13 Mar 2012 22:21:07 +0000 (15:21 -0700)]
h264: increase reference poc list from 16 to 32.

Interlaced images can have 32 references (16 per field), so limiting the
array size to 16 leads to invalid writes.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 48cbe4b092113eae0b3e5d6a08b59027f913a884)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoh264: stricter reference limit enforcement.
Ronald S. Bultje [Tue, 13 Mar 2012 23:26:44 +0000 (16:26 -0700)]
h264: stricter reference limit enforcement.

Progressive images can have only 16 references, error out if there are
more, since the data is almost certainly corrupt, and the invalid value
will lead to random crashes or invalid writes later on.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit e0febda22d0e0fab094a9c886b0e0f0f662df1ef)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoh264: improve parsing of broken AVC SPS
Michael Niedermayer [Sat, 1 Oct 2011 15:41:28 +0000 (17:41 +0200)]
h264: improve parsing of broken AVC SPS

Parsing the entire NAL as SPS fixes decoding of some AVC bitstreams
with broken escaping. Since the size of the NAL unit is known and
checked against the buffer end we can parse it entirely without buffer
overreads.

Fixes playback of
http://streams.videolan.org/streams/mp4/Mr_MrsSmith-h264_aac.mp4

Signed-off-by: Janne Grunau <janne-libav@jannau.net>
(cherry picked from commit 3aa661ec561d7a20812b84b353b0d7855ac346c8)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoReplace computations of remaining bits with calls to get_bits_left().
Alex Converse [Mon, 5 Mar 2012 01:53:50 +0000 (17:53 -0800)]
Replace computations of remaining bits with calls to get_bits_left().

(cherry picked from commit 3574a85ce57366ba7429edef93d5cad8640fb68c)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agopng: convert to bytestream2 API.
Ronald S. Bultje [Thu, 8 Mar 2012 00:16:20 +0000 (16:16 -0800)]
png: convert to bytestream2 API.

Protects against overreads in the input buffer.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 4c25269cedd042abcb823c42d33609564861c374)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoroqvideo: convert to bytestream2 API.
Ronald S. Bultje [Tue, 6 Mar 2012 23:58:35 +0000 (15:58 -0800)]
roqvideo: convert to bytestream2 API.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit cdf15771621bce7959b3e53b21426c5ba747e17b)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agosmc: port to bytestream2 API.
Ronald S. Bultje [Wed, 29 Feb 2012 22:44:37 +0000 (14:44 -0800)]
smc: port to bytestream2 API.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 8febcb9fc178926687ee19d32d2b3150da899867)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agotgq: convert to bytestream2 API.
Ronald S. Bultje [Tue, 6 Mar 2012 22:18:32 +0000 (14:18 -0800)]
tgq: convert to bytestream2 API.

This protects against input buffer overreads.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 1255eed533b4069db7f205601953ca54c0dc42c9)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoalgmm: convert to bytestream2 API.
Ronald S. Bultje [Tue, 6 Mar 2012 23:15:42 +0000 (15:15 -0800)]
algmm: convert to bytestream2 API.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit a55d5bdc6e28a2cfefc440d792de5cc4f02377e2)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agojvdec: unbreak video decoding
Paul B Mahol [Wed, 14 Mar 2012 03:02:02 +0000 (03:02 +0000)]
jvdec: unbreak video decoding

The safe bitstream reader broke it since the buffer size was specified
in bytes instead of bits.

Signed-off-by: Janne Grunau <janne-libav@jannau.net>
CC: libav-stable@libav.org
(cherry picked from commit a1c036e961a32f7208e7315dabfa0ee99d779edb)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoh264: Fix invalid interlaced/progressive MB combinations for direct mode prediction.
Michael Niedermayer [Tue, 13 Mar 2012 01:26:50 +0000 (18:26 -0700)]
h264: Fix invalid interlaced/progressive MB combinations for direct mode prediction.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
Signed-off-by: Ronald S. Bultje <rsbultje@gmail.com>
(cherry picked from commit 758ec111538ccd487686e8677aa754ee4d82beaa)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agolibx264: add 'stats' private option for setting 2pass stats filename.
Anton Khirnov [Mon, 12 Mar 2012 16:20:20 +0000 (17:20 +0100)]
libx264: add 'stats' private option for setting 2pass stats filename.

x264 always opens the file itself with fopen, so we cannot use the
standard lavc stats mechanism.

CC: libav-stable@libav.org
(cherry picked from commit d533e395e14d403948ca2424efbcee92429ef8e1)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agolibx264: fix help text for slice-max-size option.
Anton Khirnov [Mon, 12 Mar 2012 16:09:22 +0000 (17:09 +0100)]
libx264: fix help text for slice-max-size option.

CC: libav-stable@libav.org
(cherry picked from commit 9d5c131ecec75fcfb1b4b56f74f2b2756bf0027a)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoavconv: reindent
Anton Khirnov [Mon, 12 Mar 2012 16:43:48 +0000 (17:43 +0100)]
avconv: reindent

CC: libav-stable@libav.org
(cherry picked from commit 64334ddbbc7fce490c895c54106291d0b128e830)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoavconv: link '-passlogfile' option to libx264 'stats' AVOption.
Anton Khirnov [Mon, 12 Mar 2012 16:42:57 +0000 (17:42 +0100)]
avconv: link '-passlogfile' option to libx264 'stats' AVOption.

Fixes bug 204.

CC: libav-stable@libav.org
(cherry picked from commit 6e8be949f12734f38d360aad0f5c503a0f9606fa)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoRevert "h264: clear trailing bits in partially parsed NAL units"
Janne Grunau [Mon, 12 Mar 2012 21:01:02 +0000 (22:01 +0100)]
Revert "h264: clear trailing bits in partially parsed NAL units"

This reverts commit 729ebb2f185244b0ff06d48edbbbbb02ceb4ed4e.

There was an off-by-one error in the bit mask calculation clearing
actually the last valid bit and causing
http://bugzilla.libav.org/show_bug.cgi?id=227

The broken sample (Mr_MrsSmith-h264_aac.mp4) the commit was fixing
does not work after correcting the off-by-one error.

CC: libav-stable@libav.org
(cherry picked from commit 8a6037c3900875ccab8d553d2cc659bdef2c9d0e)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agompc: pad mpc_CC/SCF[] tables to allow for negative indices.
Ronald S. Bultje [Sat, 10 Mar 2012 22:28:08 +0000 (14:28 -0800)]
mpc: pad mpc_CC/SCF[] tables to allow for negative indices.

MPC8 allows indices of mpc_CC up to -1, and mpc_SCF up to -6, thus pad
the tables by that much on the left end.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit d7eabd50425a61b31e90c763a0c3e4316a725404)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoxxan: protect against chroma LUT overreads.
Ronald S. Bultje [Sat, 10 Mar 2012 19:57:17 +0000 (11:57 -0800)]
xxan: protect against chroma LUT overreads.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit f77bfa837636a99a4034d31916a76f7d1688cf5a)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoxxan: convert to bytestream2 API.
Ronald S. Bultje [Fri, 9 Mar 2012 00:32:47 +0000 (16:32 -0800)]
xxan: convert to bytestream2 API.

Protects against overreads.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 55188278169c3a1838334d7aa47a1f7a40741690)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoxxan: don't read before start of buffer in av_memcpy_backptr().
Ronald S. Bultje [Fri, 9 Mar 2012 00:32:46 +0000 (16:32 -0800)]
xxan: don't read before start of buffer in av_memcpy_backptr().

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit f1279e286b00e99f343adb51e251f036a3df6f32)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agodsicinvideo: validate buffer offset before copying pixels.
Ronald S. Bultje [Sun, 11 Mar 2012 14:28:54 +0000 (07:28 -0700)]
dsicinvideo: validate buffer offset before copying pixels.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit c95fefa0420be9cc0f09a95041acf11114aaacd0)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agocook: error out on quant_index values outside [-63, 63] range.
Ronald S. Bultje [Sun, 11 Mar 2012 01:51:28 +0000 (17:51 -0800)]
cook: error out on quant_index values outside [-63, 63] range.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 97e48b2f541396ef6e8816a555bac1bb993d7a6a)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agocook: extend channel uncoupling tables so the full bit range is covered.
Ronald S. Bultje [Tue, 6 Mar 2012 21:45:32 +0000 (13:45 -0800)]
cook: extend channel uncoupling tables so the full bit range is covered.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 37cc8600d0313838cab5b886b9d373e5819aa24f)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agocook: expand dither_tab[], and make sure indexes into it don't overflow.
Ronald S. Bultje [Fri, 9 Mar 2012 01:09:27 +0000 (17:09 -0800)]
cook: expand dither_tab[], and make sure indexes into it don't overflow.

Fixes overflows in accessing dither_tab[].

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 442c3a8cb1785d74f8e2d7ab35b1862b7088436b)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agohuffyuv: add padding to classic (v1) huffman tables.
Ronald S. Bultje [Thu, 8 Mar 2012 00:29:23 +0000 (16:29 -0800)]
huffyuv: add padding to classic (v1) huffman tables.

We slightly overread the input buffer, so we require
padding at the end of the buffer, as is documented in the
get_bits API. Without padding, we'll read uninitialized
data or beyond the end of the .rodata, which may crash.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 4ffe5e2aa5241f8da9afd2c8fbc854dcc916c5f9)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoavs: fix infinite loop on end-of-stream.
Ronald S. Bultje [Thu, 16 Feb 2012 00:21:34 +0000 (16:21 -0800)]
avs: fix infinite loop on end-of-stream.

The codec would keep returning the last decoded frame if the stream
contains B-frames, since it wouldn't clear that frame from the list of
frames to be returned to the user.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 83f15a1228895434a982c840b09edccd1c64e800)

Conflicts:

libavcodec/cavsdec.c

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agotiffdec: Prevent illegal memory access caused by recycled pointers.
Alex Converse [Wed, 7 Mar 2012 01:00:29 +0000 (17:00 -0800)]
tiffdec: Prevent illegal memory access caused by recycled pointers.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit fd0be63049ed46660993d0550a4f0847a0b942ea)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agowma: fix off-by-one in array bounds check.
Ronald S. Bultje [Wed, 7 Mar 2012 22:18:14 +0000 (14:18 -0800)]
wma: fix off-by-one in array bounds check.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit b4bccf3e4e58f6fe58043791ca09db01a4343fac)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agodv: check buffer size before reading profile.
Ronald S. Bultje [Wed, 7 Mar 2012 21:48:41 +0000 (13:48 -0800)]
dv: check buffer size before reading profile.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit e97efecec82ca8458a9bbd75a91ebf556abde362)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoraw: move buffer size check up.
Ronald S. Bultje [Wed, 7 Mar 2012 00:08:10 +0000 (16:08 -0800)]
raw: move buffer size check up.

This way, it protects against overreads for 4bpp/2bpp content also.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit cc5dd632cecc5114717d0b90f8c2be162b1c6ee8)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agodca: prevent accessing static arrays with invalid indexes.
Ronald S. Bultje [Wed, 29 Feb 2012 02:11:59 +0000 (18:11 -0800)]
dca: prevent accessing static arrays with invalid indexes.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit e6ffd997cbc06426e75d3fa291b991866c84a79b)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agolpcm: fix sample size calculation for 20bit LCPM.
Ronald S. Bultje [Wed, 7 Mar 2012 04:08:17 +0000 (20:08 -0800)]
lpcm: fix sample size calculation for 20bit LCPM.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit f1320dc3bed281bb2f3c5531c52b6a6246e2394a)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agosmacker: error out if palette copy-with-offset overruns palette size.
Ronald S. Bultje [Wed, 7 Mar 2012 01:24:20 +0000 (17:24 -0800)]
smacker: error out if palette copy-with-offset overruns palette size.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit a93b572ae4f517ce0c35cf085167c318e9215908)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoDon't use ff_cropTbl[] for IDCT.
Ronald S. Bultje [Tue, 6 Mar 2012 00:01:19 +0000 (16:01 -0800)]
Don't use ff_cropTbl[] for IDCT.

Results of IDCT can by far outreach the range of ff_cropTbl[], leading
to overreads and potentially crashes.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit c23acbaed40101c677dfcfbbfe0d2c230a8e8f44)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoswscale: make filterPos 32bit.
Ronald S. Bultje [Mon, 5 Mar 2012 20:26:42 +0000 (12:26 -0800)]
swscale: make filterPos 32bit.

Fixes overflows for large image sizes.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 2254b559cbcfc0418135f09add37c0a5866b1981)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoerror_resilience: initialize s->block_index[].
Ronald S. Bultje [Tue, 6 Mar 2012 18:27:05 +0000 (10:27 -0800)]
error_resilience: initialize s->block_index[].

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 6193ff68549ecbaf1a4d63a0e06964ec580ac620)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agosvq3: protect against negative quantizers.
Ronald S. Bultje [Tue, 6 Mar 2012 01:03:32 +0000 (17:03 -0800)]
svq3: protect against negative quantizers.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 11b940a1a8e7e5d5b212935a3ce78aeda577f5f2)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agoPrepare for 0.8.1 Release
Reinhard Tartler [Mon, 5 Mar 2012 19:40:37 +0000 (20:40 +0100)]
Prepare for 0.8.1 Release

7 years agomov: set channel layout for AC-3 streams based on the 'dac3' atom info
Justin Ruggles [Sun, 12 Feb 2012 20:06:58 +0000 (15:06 -0500)]
mov: set channel layout for AC-3 streams based on the 'dac3' atom info

fixes Bug 225
(cherry picked from commit 3798205a77ce275613098ecb48645e6029811f14)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agorv34: handle size changes during frame multithreading
Janne Grunau [Mon, 13 Feb 2012 20:10:48 +0000 (21:10 +0100)]
rv34: handle size changes during frame multithreading

Factors all context dynamic memory handling to its own functions.
Fixes bug 220.
(cherry picked from commit 2bd730010da24d035639586bb13862abe36cc1b8)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
7 years agomov: Add more HDV and XDCAM FourCCs.
Alex Converse [Tue, 21 Feb 2012 23:37:35 +0000 (15:37 -0800)]
mov: Add more HDV and XDCAM FourCCs.

Reference: VLC
(cherry picked from commit b142496c5630b9bc88fb9eaccae7f6bd62fb23e7)

7 years agomov: Add support for MPEG2 HDV 720p24 (hdv4)
Alex Converse [Tue, 21 Feb 2012 22:08:02 +0000 (14:08 -0800)]
mov: Add support for MPEG2 HDV 720p24 (hdv4)

(cherry picked from commit 0ad522afb3a3b3d22402ecb82dd4609f7655031b)

7 years agorv10/20: Fix slice overflow with checked bitstream reader.
Alex Converse [Thu, 1 Mar 2012 21:24:55 +0000 (13:24 -0800)]
rv10/20: Fix slice overflow with checked bitstream reader.

(cherry picked from commit 9243ec4a508c81a621e941bb7e012e2d45d93659)

7 years agoh263dec: Disallow width/height changing with frame threads.
Michael Niedermayer [Fri, 17 Feb 2012 21:35:10 +0000 (13:35 -0800)]
h263dec: Disallow width/height changing with frame threads.

Fixes CVE-2011-3937

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 71db86d53b5c6872cea31bf714a1a38ec78feaba)

Conflicts:

libavcodec/h263dec.c

Signed-off-by: Alex Converse <alex.converse@gmail.com>
7 years agoadpcm: Clip step_index values read from the bitstream at the beginning of each frame.
Alex Converse [Tue, 28 Feb 2012 19:50:22 +0000 (11:50 -0800)]
adpcm: Clip step_index values read from the bitstream at the beginning of each frame.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
(cherry picked from commit bbeb29133b55b7256d18f5aaab8b5c8e919a173a)

7 years agotiff: Make the TIFF_LONG and TIFF_SHORT types unsigned.
Alex Converse [Thu, 23 Feb 2012 18:22:51 +0000 (10:22 -0800)]
tiff: Make the TIFF_LONG and TIFF_SHORT types unsigned.

TIFF v6.0 (unimplemented) adds signed equivalents.
(cherry picked from commit e32548d1331ce05a054f1028fcdda8823a4f215a)

7 years agodpcm: ignore extra unpaired bytes in stereo streams.
Alex Converse [Fri, 17 Feb 2012 22:13:40 +0000 (14:13 -0800)]
dpcm: ignore extra unpaired bytes in stereo streams.

Fixes: CVE-2011-3951

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
(cherry picked from commit ce7aee9b733134649a6ce2fa743e51733f33e67e)

7 years agosvq3: Prevent illegal reads while parsing extradata.
Alex Converse [Fri, 10 Feb 2012 04:21:47 +0000 (20:21 -0800)]
svq3: Prevent illegal reads while parsing extradata.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
(cherry picked from commit 9e1db721c4329f4ac166a0bcc002c8d75f831aba)

7 years agodv: Fix small overread in audio frequency table.
Alex Converse [Fri, 10 Feb 2012 01:11:55 +0000 (17:11 -0800)]
dv: Fix small overread in audio frequency table.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
(cherry picked from commit 0ab3687924457cb4fd81897bd39ab3cc5b699588)

7 years agoac3dec: Move center and surround mix level tables to the parser.
Michael Niedermayer [Fri, 3 Feb 2012 03:27:27 +0000 (22:27 -0500)]
ac3dec: Move center and surround mix level tables to the parser.

That way all mix levels as exported by avpriv_ac3_parse_header()
will have the same meaning.

Previously the 3-bit center mix level for E-AC-3 was used to index in a
4-entry table, leading to out-of-array reads.

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Signed-off-by: Justin Ruggles <justin.ruggles@gmail.com>
Signed-off-by: Alex Converse <alex.converse@gmail.com>
(cherry picked from commit e6d9fa66f12cf5a3024c9bc7c4c608f7fc59207e)

7 years agomovdec: Avoid av_malloc(0) in stss
Alex Converse [Fri, 3 Feb 2012 18:43:21 +0000 (10:43 -0800)]
movdec: Avoid av_malloc(0) in stss

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
(cherry picked from commit 29a20ac4a19df5acc0eef306ca5a737778a31358)

7 years agoac3: Do not read past the end of ff_ac3_band_start_tab.
Mans Rullgard [Tue, 31 Jan 2012 18:20:33 +0000 (10:20 -0800)]
ac3: Do not read past the end of ff_ac3_band_start_tab.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Alex Converse <alex.converse@gmail.com>
(cherry picked from commit 034b03e7a0e8e4f8f66c82b736f2c0aa7c063ec0)

7 years agodv: Fix small stack overread related to CVE-2011-3929 and CVE-2011-3936.
Alex Converse [Thu, 26 Jan 2012 23:08:26 +0000 (15:08 -0800)]
dv: Fix small stack overread related to CVE-2011-3929 and CVE-2011-3936.

Found with asan.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Alex Converse <alex.converse@gmail.com>
(cherry picked from commit 2d1c0dea5f6b91bec7f5fa53ec050913d851e366)

7 years agodv: Fix null pointer dereference due to ach=0
Michael Niedermayer [Tue, 24 Jan 2012 16:51:40 +0000 (17:51 +0100)]
dv: Fix null pointer dereference due to ach=0

dv: Fix null pointer dereference due to ach=0

Fixes part2 of CVE-2011-3929

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Reviewed-by: Roman Shaposhnik <roman@shaposhnik.org>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Signed-off-by: Alex Converse <alex.converse@gmail.com>
(cherry picked from commit 5a396bb3a66a61a68b80f2369d0249729bf85e04)

7 years agodv: check stype
Michael Niedermayer [Tue, 24 Jan 2012 16:48:23 +0000 (17:48 +0100)]
dv: check stype

dv: check stype

Fixes part1 of CVE-2011-3929
Possibly fixes part of CVE-2011-3936

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Reviewed-by: Roman Shaposhnik <roman@shaposhnik.org>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Signed-off-by: Alex Converse <alex.converse@gmail.com>
(cherry picked from commit 635bcfccd439480003b74a665b5aa7c872c1ad6b)

7 years agompegaudiodec: Prevent premature clipping of mp3 input buffer.
Dale Curtis [Fri, 24 Feb 2012 18:17:39 +0000 (13:17 -0500)]
mpegaudiodec: Prevent premature clipping of mp3 input buffer.

Instead of clipping extrasize based on EXTRABYTES, clip based on the
amount of buffer actually left. Without this fix, there are warbles
and other distortions in the test case below.

http://kevincennis.com/mix/assets/sounds/1901_voxfx.mp3
(cherry picked from commit b7165426917f91ebcad84bdff366824f03b32bfe)

Signed-off-by: Alex Converse <alex.converse@gmail.com>
7 years agomp3dec: Fix a heap-buffer-overflow
Alex Converse [Wed, 25 Jan 2012 23:46:14 +0000 (15:46 -0800)]
mp3dec: Fix a heap-buffer-overflow

In some cases, what is left to read from ptr is smaller than EXTRABYTES.

Based on a patch by Thierry Foucu <tfoucu@gmail.com>.

Signed-off-by: Alex Converse <alex.converse@gmail.com>
(cherry picked from commit f372ce119bd2458fa0b4ddfb2af3a36621df99f7)

7 years agompeg12: Pad framerate tab to 16 entries.
Alex Converse [Fri, 27 Jan 2012 23:50:24 +0000 (15:50 -0800)]
mpeg12: Pad framerate tab to 16 entries.

There are many places where we read an unchecked 4-bit index into it.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
(cherry picked from commit dfa37fe8a3d9243dd339d94befa065e2c90b29e6)

7 years agokgv1dec: Increase offsets array size so it is large enough.
Michael Niedermayer [Wed, 25 Jan 2012 22:23:35 +0000 (23:23 +0100)]
kgv1dec: Increase offsets array size so it is large enough.

Fixes CVE-2011-3945

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 807a045ab7f51993a2c1b3116016cbbd4f3d20d6)

Signed-off-by: Alex Converse <alex.converse@gmail.com>
(cherry picked from commit a02e8df973f5478ec82f4c507f5b5b191a5ecb6b)

7 years agokmvc: Check palsize.
Alex Converse [Thu, 26 Jan 2012 16:30:49 +0000 (17:30 +0100)]
kmvc: Check palsize.

Fixes: CVE-2011-3952

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Based on fix by Michael Niedermayer
(cherry picked from commit 386741f887714d3e46c9e8fe577e326a7964037b)

7 years agonsvdec: Propagate errors
Alex Converse [Fri, 27 Jan 2012 01:23:09 +0000 (17:23 -0800)]
nsvdec: Propagate errors

Related to CVE-2011-3940.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
(cherry picked from commit c898431ca5ef2a997fe9388b650f658fb60783e5)

Conflicts:

libavformat/nsvdec.c

7 years agonsvdec: Be more careful with av_malloc().
Alex Converse [Fri, 27 Jan 2012 01:21:46 +0000 (17:21 -0800)]
nsvdec: Be more careful with av_malloc().

Check results for av_malloc() and fix an overflow in one call.

Related to CVE-2011-3940.

Based in part on work from Michael Niedermayer.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
(cherry picked from commit 8fd8a48263ff1437f9d02d7e78dc63efb9b5ed3a)

7 years agonsvdec: Fix use of uninitialized streams.
Michael Niedermayer [Tue, 24 Jan 2012 21:20:26 +0000 (22:20 +0100)]
nsvdec: Fix use of uninitialized streams.

Fixes CVE-2011-3940 (Out of bounds read resulting in out of bounds write)

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 5c011706bc752d34bc6ada31d7df2ca0c9af7c6b)

Signed-off-by: Alex Converse <alex.converse@gmail.com>
(cherry picked from commit 6a89b41d9780325ba6d89a37f2aeb925aa68e6a3)

7 years agog722: Fix the QMF scaling
Martin Storsjö [Fri, 2 Mar 2012 15:03:06 +0000 (17:03 +0200)]
g722: Fix the QMF scaling

This fixes clipping if the encoder input used the full 16 bit
input range (samples with a magnitude below 16383 worked fine).
The filtered subband samples should be 15 bit maximum, while
the code earlier produced them scaled to 16 bit.

This makes the decoder output have double the magnitude
compared to before.

The spec reference samples doesn't test the QMF at all, which
was why this part slipped past initially.

(cherry picked from commit b087ce2bee81db8cc5caffb8f0a4f6c7c92a30fe)

Signed-off-by: Martin Storsjö <martin@martin.st>