lavc/h264_mp4toannexb: improve feedback in case of invalid bitstream