rtpenc: fix overflow checking in avc_mp4_find_startcode()
authorXi Wang <xi.wang@gmail.com>
Wed, 23 Jan 2013 01:58:07 +0000 (20:58 -0500)
committerMichael Niedermayer <michaelni@gmx.at>
Thu, 7 Feb 2013 00:00:01 +0000 (01:00 +0100)
commitbc58fe0309a075b6f75aeef948a55eede927591c
tree9428ca7a16ab7cc9f20432b48a579981cda85788
parent8eda88868399de00806cf21a966d9660db4ae9b4
rtpenc: fix overflow checking in avc_mp4_find_startcode()

The check `start + res < start' is broken since pointer overflow is
undefined behavior in C.  Many compilers such as gcc/clang optimize
away this check.

Use `res > end - start' instead.  Also change `res' to unsigned int
to avoid signed left-shift overflow.

Signed-off-by: Xi Wang <xi.wang@gmail.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 2f014567cfd63e58156f60666f1a61ba147276ab)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
libavformat/rtpenc_h264.c