avcodec/realtextdec: add some memory checks