rmenc: Drop the temporary buffer for ac3 byteswap