avutil/mathematics: Do not treat INT64_MIN as positive in av_rescale_rnd