swscale: enforce a minimum filtersize.
authorRonald S. Bultje <rsbultje@gmail.com>
Sat, 11 Feb 2012 16:42:28 +0000 (08:42 -0800)
committerRonald S. Bultje <rsbultje@gmail.com>
Sat, 11 Feb 2012 18:06:50 +0000 (10:06 -0800)
At very small dimensions, this calculation could lead to zero-sized
filters, which leads to uninitialized output, zero-sized allocations,
loop overflows in SIMD that uses do{..}while(i++<filtersize); instead
of for(i=0;i<filtersize;i++){..} and several other similar failures.
Therefore, require a minimum filtersize of 1.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
libswscale/utils.c

index 15039ce1f028b90be0a6aea569126337905fa0da..2fe9c5b47f18122256a05fe042f20a6b17d896bc 100644 (file)
@@ -263,7 +263,7 @@ static int initFilter(int16_t **outFilter, int16_t **filterPos, int *outFilterSi
         if (xInc <= 1<<16)      filterSize= 1 + sizeFactor; // upscale
         else                    filterSize= 1 + (sizeFactor*srcW + dstW - 1)/ dstW;
 
-        if (filterSize > srcW-2) filterSize=srcW-2;
+        filterSize = av_clip(filterSize, 1, srcW - 2);
 
         FF_ALLOC_OR_GOTO(NULL, filter, dstW*sizeof(*filter)*filterSize, fail);