lavu/opt: fix range check logic in set_format()