avcodec/tiff: Check subsample & rps values more completely