avfilter/vf_maskedmerge: add support for more pixel formats