vp9: change type of tile_size from unsigned to int64_t
authorAndreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
Sun, 7 Jun 2015 16:50:43 +0000 (18:50 +0200)
committerAndreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
Mon, 8 Jun 2015 19:43:16 +0000 (21:43 +0200)
Otherwise the check 'tile_size < size' treats a negative size as
unsigned, causing the check to pass. This subsequently leads to
segmentation faults.

This was originally fixed as part of Libav commit 72ca83, so the
original author is one of the following developers:
        Anton Khirnov <anton@khirnov.net>
        Diego Biurrun <diego@biurrun.de>
        Luca Barbato <lu_zero@gentoo.org>
        Martin Storsj√∂ <martin@martin.st>

Reviewed-by: Ronald S. Bultje <rsbultje@gmail.com>
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
libavcodec/vp9.c

index c4efd42baf3a6c89ded5f1ed9d65a2d7385cea41..d5147e5b044de47c8343f5756b25e840e072615e 100644 (file)
@@ -4106,7 +4106,7 @@ static int vp9_decode_frame(AVCodecContext *ctx, void *frame,
                             tile_row, s->tiling.log2_tile_rows, s->sb_rows);
             if (s->pass != 2) {
                 for (tile_col = 0; tile_col < s->tiling.tile_cols; tile_col++) {
-                    unsigned tile_size;
+                    int64_t tile_size;
 
                     if (tile_col == s->tiling.tile_cols - 1 &&
                         tile_row == s->tiling.tile_rows - 1) {