avcodec/mpegvideo_enc: return proper error instead of failing assertion when max...