avformat/utils: prevent r frame rate from being set larger than 1/tb