avformat/utils: factor rfps calculation out