swscale: fix invalid conversions and memory problems.