avcodec/iff: finetune the palette size check in the mask case