avcodec/dxtory: fix src size checks