bitstream: Check the result of av_malloc()