postproc/postprocess: Check for av_mallocz() failure