avcodec/vc1: Check for excessive resolution