golomb: always check for invalid UE golomb codes in get_ue_golomb
authorAndreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
Sun, 13 Dec 2015 20:02:16 +0000 (21:02 +0100)
committerAndreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
Thu, 17 Dec 2015 20:37:19 +0000 (21:37 +0100)
commit1828abb7449a06c71a6f1199efc9b5e480502603
tree41015d24a3a3e9a704a0ce3a565637c81b660ff7
parent4733060ef39466a443680e4f453e06d375844294
golomb: always check for invalid UE golomb codes in get_ue_golomb

Also correct the check to reject log < 7, because UPDATE_CACHE only
guarantees 25 meaningful bits.

This fixes undefined behavior:
runtime error: shift exponent is negative

Testing with START/STOP timers in get_ue_golomb, one for the first
branch (A) and one for the second (B), shows that there is practically no
slowdown, e.g. for the cavs decoder:

With the check in the B branch:
    629 decicycles in get_ue_golomb B, 4194260 runs,     44 skips
    433 decicycles in get_ue_golomb A,268434102 runs,   1354 skips

Without the check:
    624 decicycles in get_ue_golomb B, 4194273 runs,     31 skips
    433 decicycles in get_ue_golomb A,268434203 runs,   1253 skips

Since the B branch is executed far less often than the A branch, this
change is negligible, even more so for the h264 decoder, where the ratio
B/A is a lot smaller.

Fixes: mozilla bug 1230239
Fixes: fbeb8b2c7c996e9b91c6b1af319d7ebc/asan_heap-oob_195450f_2743_e8856ece4579ea486670be2b236099a0.bit

Found-by: Tyson Smith
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
(cherry picked from commit 22e960ad478e568f4094971a58c6ad8f549c0180)
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
libavcodec/golomb.h