avcodec/mss2: Check for repeat overflow