avformat/movenc: Fix potential signed overflows