avutil/libm: fix isnan compatibility hack