avcodec/h264: more completely check the loop filter parameters