transform: give avfilter_transform() a return value for returning an error