avcodec/xan: Improve overlapping check