dnxhdenc: fix signed overflows