avcodec/xsubdec: fix overflow in alpha handling